Hybrid RAG for Legal Text: Chunking, BM25, BGE and Reranking

Legal text is a hard test for any retrieval system. Take the Constitution of India: it has hundreds of articles and clauses, plus provisos, schedules, amendments, tables and a web of cross-references. The passage answering a question may be one sentence, yet reading it correctly can depend on the surrounding article, a related exception or another provision it points to. A useful system therefore has to balance precision and context, finding the exact passage without cutting it off from the text that gives it meaning.

Many introductory retrieval-augmented generation (RAG) setups follow a simple recipe: extract text from a PDF, cut it into fixed-size chunks, embed them and return the closest matches. That's a fair starting point, but production systems usually need more. They have to keep document structure, support both exact and semantic search, rerank candidates, recover surrounding context, trace every result to its source and measure whether the needed evidence was really retrieved.

This write-up describes a research project that reproduces such a production-style workflow in a form that can be inspected stage by stage, using the Constitution of India as the first corpus because its structure exposes so many weaknesses of plain vector search. It isn't presented as a finished product. It's an attempt to study how the usual components of a production RAG pipeline fit together, where they break and how evaluation can steer the next design decision.

1. Ingestion: from legal PDF to structured content

Before chunking, embeddings or retrieval can be considered, the source PDF has to become content that software can process reliably. A PDF records how a document looks, not necessarily how a program should read it. Headings, clauses, footnotes, lists, tables and page boundaries may be obvious to a person and still come out of basic extraction as disconnected text. Lose that structure at ingestion and later stages produce chunks with wrong reading order, detached headings or broken tables.

The project used MinerU to convert the Constitution into Markdown while keeping the important layout information. It was picked because, in the author's comparison, it handled tables and lists more reliably than the other extraction approach tried. The goal wasn't just to collect every word but to preserve the relationships between the words. The extracted content kept:

• how headings nest,
• where each paragraph and clause starts and ends,
• bulleted and numbered lists,
• tables,
• which page each piece came from, and
• footnotes along with the structure around them.

Markdown suited this role as an intermediate format. People can read it during manual inspection, and it carries structural markers that Python can process. A heading such as ## Fundamental Rights can later guide the creation of parent chunks, while lists and tables can be recognized and treated as separate blocks.

The extraction ran in Google Colab and not on a local machine. OCR and layout analysis can be heavy, particularly for a long document full of tables and complex formatting, and a laptop with limited hardware struggled with the workload. Colab offered a better-suited isolated environment with stronger compute, so it's worth considering for this stage if your local hardware is modest. The command used in the notebook was:

!mineru -p c.pdf -o output --dump-content-list -b pipeline -l en

2. Chunking

With the Constitution in structured Markdown, the next question was how to divide it for retrieval. The common answer is fixed-size chunking: cut after a set number of characters or tokens, often with some overlap. It's simple and works acceptably for uniform prose, but legal documents are organized by meaning and not by length. A fixed boundary can:

• separate a heading from the article under it,
• cut a clause off from the proviso that qualifies it,
• slice a list through the middle of its entries,
• break a table into incomplete fragments, and
• merge unrelated articles just because they fit in one token window.

Bigger chunks keep more context but make retrieval blurrier. Smaller chunks match better, yet important surrounding conditions may vanish. The project therefore used a two-level strategy:

Structured document
        ↓
Heading-aware parent chunks
        ↓
Semantic child chunks

Why two levels help

Search happens over the small children, because their narrow content gives BM25 and dense retrieval cleaner signals. After a child is picked, the parent_id it carries points to the larger parent section:

Query
  ↓
Retrieve focused child
  ↓
Read parent_id
  ↓
Return complete parent context

That splits two responsibilities. Child chunks supply retrieval precision, and parent chunks supply completeness of context.

Heading-aware parent chunks

The parent-chunking algorithm relies on two artifacts produced during ingestion:

data/processed/cr.md
data/extracted/auto/blockpage.json

The Markdown file holds the cleaned text and its visible structure. The blockpage.json file holds the ordered layout blocks generated during extraction, including heading-level information. Markdown remains the main source of parent text, and the JSON works as an ordered structural reference for matching Markdown headings back to the extracted document. The implementation lives in a single script:

code/chunking_parent.py

The algorithm works in a fixed order.

Step 1: load both artifacts. The script reads in the cleaned Markdown and also the JSON file of layout blocks:

md_text = MARKDOWN_PATH.read_text(encoding="utf-8")

with BLOCKPAGE_PATH.open("r", encoding="utf-8") as file:
    blockpage = json.load(file)

Using both stops the chunker from treating the Markdown as plain, unstructured text. The Markdown supplies content, and the JSON helps keep the order of the headings found during extraction.

Step 2: split the Markdown into blocks. Blank lines are the separator:

for raw_block in md_text.split("\n\n"):

Each block is then classified into one of several types:

heading
paragraph
list
table
superscript
blank

Text that starts with #, ## or another heading level, for example, is recognized as a heading:

HEADING_RE = re.compile(r"^(#{1,6})\s+")

Lists, tables and superscript blocks get their own detection rules, which keeps them tied to the section they belong to.

Step 3: track the active heading. When a heading appears, its text is added to the current heading_path. The script also looks up the matching extracted heading in blockpage.json, moving through the JSON sequentially and not rescanning from the beginning each time:

def find_heading(heading_text: str):
    nonlocal search_pos

    for index in range(search_pos, len(blockpage)):
        block = blockpage[index]        if (
            block.get("type") == "text"
            and block.get("text") == heading_text
            and "text_level" in block
        ):
            search_pos = index + 1
            return block

Scanning forward only keeps headings in their original order and prevents a hit on an earlier, identical heading.

Step 4: collect related content. Paragraphs, lists, tables and similar blocks are added to the current parent stack. When a new heading arrives after body text has already been collected, the current parent is closed and another begins. Conceptually it looks like this:

Heading
  ├── Paragraph
  ├── Clause
  ├── List
  └── Table
        ↓
     Parent chunk

Step 5: assign stable metadata. Each finished parent is stored with a sequential ID, a document ID, the heading path and the full text:

{
    "parent_id": f"parent_{len(chunks):04d}",
    "doc_id": DOC_ID,
    "heading_path": heading_path.copy(),
    "text": "\n\n".join(stack),
}

A simplified parent looks like this:

{
  "parent_id": "parent_0164",
  "doc_id": "constitution_of_india",
  "heading_path": ["PART XII"],
  "text": "# PART XII\n\nFINANCE, PROPERTY, CONTRACTS AND SUITS..."
}

A single parent may hold several related provisions, for instance:

PART XII — Finance

Article 264 — Interpretation
Article 265 — Taxes not to be imposed without authority of law
Article 266 — Consolidated Funds and public accounts
Article 267 — Contingency Fund

The process produced 333 parent chunks.

Semantic child chunks

Parents keep context intact, yet a few are too wide for precise matching, so the bigger ones are split into smaller semantic children. That work lives in a notebook:

code/semantic-children.ipynb

Short parents, judged by the notebook's rough 400-token threshold, are kept whole as a single child, because splitting something already compact enough to serve as a focused unit gains nothing. Anything longer is handled by a chunker that respects structure and compares meaning.

Step 1: break the parent into blocks. The parent text is separated into headings, paragraphs, lists and tables. HTML tables are recognized with their own pattern:

table_pattern = re.compile(
    r"(<table[\s\S]*?</table>)",
    re.IGNORECASE
)

Ordinary text is split on blank lines and classified by Markdown headings and list markers, so prose, lists and tables can each be handled in their own way instead of assuming one structure fits all.

Step 2: attach headings to their content. A heading should never become a child on its own. The chunker holds each heading temporarily and prepends it to the block that follows:

if pending_heading:
    block.text = pending_heading + "\n\n" + block.text
    pending_heading = None

A retrieved child thus keeps the heading that identifies its subject. Consecutive list blocks are also grouped before semantic comparison, which helps preserve the logical order of clauses and enumerated items.

Step 3: compare neighboring paragraphs. For adjacent paragraph blocks, the notebook produces BGE embeddings:

embedder = SentenceTransformer(
    "BAAI/bge-base-en-v1.5"
)

The notebook then computes cosine similarity between each paragraph and the one before it:

similarity = cosine_similarity(
    emb1,
    emb2
)[0][0]

If two blocks are close enough in meaning, they may stay in the same child, and a drop in similarity hints that the next block starts a new topic. The configured threshold is:

Similarity threshold: 0.40

Similarity is only one input, though. Block type and size are weighed too.

Step 4: apply size and structure rules. The semantic chunker's defaults are:

Preferred minimum size: 800 characters
Maximum combined size: 1,600 characters
Tiny-block threshold: 150 characters

A block can be added to the current child when the child is still under the preferred minimum, when the neighboring paragraphs are similar enough, when one of the blocks is tiny and shouldn't stand alone, or when consecutive blocks belong to the same list or table. The combined child normally has to stay within 1,600 characters. A table won't be glued onto normal prose just to hit the minimum size:

if (
    current_type == BlockType.TABLE
    and block.type != BlockType.TABLE
) or (
    block.type == BlockType.TABLE
    and current_type != BlockType.TABLE
):
    is_under_min = False

An oversized list can be cut at its item markers, whereas a single oversized non-list block is kept as one structural unit and not sliced blindly. The result is neither purely semantic nor purely size-driven chunking. It combines:

Document structure
        +
Semantic similarity
        +
Minimum and maximum sizes

Step 5: preserve the parent link. Every generated child gets its own ID and keeps the ID of its parent:

{
    "parent_id": parent["parent_id"],
    "child_id": f"{parent['parent_id']}_child_{index}",
    "doc_id": parent.get("doc_id", ""),
    "heading_path": parent.get("heading_path", []),
    "text": chunk["text"],
    "type": chunk.get("type", "mixed"),
    "embedding": encode_text(chunk["text"])
}

Using the Finance parent as a toy case:

parent_0164
│
├── parent_0164_child_1
│   Article 264 and introductory Finance context
│
├── parent_0164_child_2
│   Articles 265 and 266 concerning taxation and public funds
│
└── parent_0164_child_3
    Article 267 concerning the Contingency Fund

All three children keep:

{
  "parent_id": "parent_0164"
}

A stored child looks like this:

{
  "child_id": "parent_0164_child_2",
  "parent_id": "parent_0164",
  "doc_id": "constitution_of_india",
  "heading_path": ["PART XII"],
  "type": "list",
  "text": "265. Taxes not to be imposed save by authority of law...",
  "embedding": [0.012, -0.034, 0.021]
}

In the end the corpus holds 871 children tied back to 333 parents.

3. Hybrid retrieval, from query to parent context

With 871 children in place, the question became how to find the right evidence for a user's query. Retrieval runs on child chunks and not on parents, since short pieces line up more easily with one specific article, clause or idea. The larger parents are recovered only once the strongest children are known.

Why a single method isn't enough

Legal questions come in different forms. A query such as "What does Article 23 say?" contains an article number that also appears in the source, so keyword retrieval suits it well. A user might ask for the same thing differently: "Can the State require someone to perform compulsory service?" That wording doesn't name Article 23, so the system must connect "require someone to perform service" with the idea of compulsory service for a public purpose. The two cases call for different abilities:

• literal matching, for article numbers, legal phrases, names and unusual terms, and
• meaning-based matching, for when the wording of the question drifts from the text.

Instead of choosing, the system runs the same child corpus through two separate branches: BM25 sparse retrieval and BGE dense retrieval.

The sparse branch: BM25

BM25 ranks by the words a query shares with each child. At indexing time, heading paths and child text are lowercased and tokenized, and the index records which children contain each term, how often it occurs and how rare it is across the corpus. Three signals matter: how often a query term appears in a child, how uncommon that term is in the corpus, and the child's length compared with the average. Rare terms count for more than common words, which makes BM25 especially good with article numbers and specific legal expressions.

A search for Article 23 compulsory service, for example, strongly favors children containing 23, compulsory and service. BM25 doesn't need to grasp the broader meaning, since the direct presence of the terms is already a strong signal. Its weakness is that a relevant passage may score poorly when the query and the source use different vocabulary.

The dense branch: BGE and FAISS

The dense branch searches by meaning and not only by word overlap. Each child is represented by its heading path plus its text:

embedding_text = heading_path + child_text

Including the heading adds context, which helps related passages and queries land near one another in embedding space.

Picking the embedding model with a benchmark

The first version of the project used nomic-ai/nomic-embed-text-v1.5, but the author wanted to confirm it was the best fit for this corpus. A controlled dense-retrieval benchmark compared four setups: Nomic without prefixes, Nomic with its recommended query and document prefixes, BAAI/bge-base-en-v1.5 and intfloat/e5-base-v2.

Every model searched the identical set of 871 fixed child chunks, with 383 evaluation questions that had been mapped automatically.

• MRR: gives more credit when the first relevant child sits nearer the top.
• Recall@10 and Recall@20: fraction of questions that get a relevant child somewhere in the top 10 or 20.
• All-recall@10, the share with every relevant child in the top 10.
• Mean coverage@10: what share of the relevant children land in the top 10, averaged over questions.

BGE had the best MRR, Recall@10, Recall@20 and mean coverage@10. MRR rose from 0.624 with Nomic to 0.652, and Recall@20 climbed from 0.877 to 0.911. That mattered because later fusion and reranking use a limited candidate pool, and a relevant child missing from dense retrieval can't be recovered afterward.

Nomic scored slightly higher on All-recall@10 (0.347 against 0.334), meaning it was marginally better at retrieving every labeled child for multi-child questions. Even so, finding at least one strong child earlier counted for more in recovering the parent context. Query latency was almost identical. The stored Nomic child embeddings were therefore replaced with BGE embeddings, keeping the existing semantic boundaries so the embedding change stayed separate from earlier chunking choices.

Children are embedded as plain text with no instruction prefix. Queries, on the other hand, get the retrieval instruction BGE recommends:

query_text = (
    "Represent this sentence for searching relevant passages: "
    + user_query
)

The same BGE model embeds queries and children. Vectors are L2-normalized, so the inner product behaves like cosine similarity, with higher scores meaning greater similarity.

The FAISS index

The normalized child vectors go into a FAISS Hierarchical Navigable Small World index:

index = faiss.IndexHNSWFlat(
    embedding_dimension,
    32,
    faiss.METRIC_INNER_PRODUCT,
)

index.hnsw.efConstruction = 200
index.hnsw.efSearch = 64
index.add(embeddings)

HNSW arranges the embeddings as a navigable graph, so a search moves through promising neighbors and doesn't compare the query with every vector. An exhaustive comparison would still be manageable for 871 children. HNSW was chosen so the design can keep working when further statutes, regulations, judgments and similar documents join the corpus. The children were re-embedded with code/reembed_children_bge.py, and the index was rebuilt with code/rebuild_dense_index.py. The benchmark preparation, model definitions, queries and results sit under code/dense_benchmark/v2/.

Combining the two rankings

BM25 and dense retrieval produce separate ranked lists. A child may rank high in both, strongly in only one, or be missing from one. Raw scores can't just be added, because a BM25 score and a vector similarity live on different scales. The two rankings are merged with Reciprocal Rank Fusion:

RRF score = Σ 1 / (k + rank + 1)

The pipeline uses k = 60. Each child earns points from its position in every list and not from its original score. A child ranking high in both branches collects contributions from both and usually moves up, while one that's strong in a single branch can still stay in the candidate set. The simplified fusion stage:

sparse_results = bm25_search(query)
dense_results = dense_search(query)

fused_candidates = reciprocal_rank_fusion(
    sparse_results,
    dense_results,
)

RRF produces a broad candidate pool. It improves the odds that useful evidence survives, but it doesn't compare the whole query against each passage in detail. That is the reranker's job.

Cross-encoder reranking

Dense search is cheap since the query and the children are encoded separately, so child embeddings can be computed once and reused for every query. The price is that independent encoding caps how closely the model can weigh their words against each other. A cross-encoder takes another route and receives the query and a candidate child together:

pairs = [
    (query, child_text),
    ...
]

scores = reranker.predict(pairs)

Since the query and passage go in as a single input, the model can look at how their tokens relate to each other directly, and that generally gives a finer relevance verdict than the positions coming from BM25 or dense search.

candidates = fused_candidates[:30]
reranked_children = cross_encoder.rank(query, candidates)

The reranker scores each child anew and sorts the candidates again. The earlier BM25, dense and RRF positions only generate the pool, and the cross-encoder score decides the final order. The system then looks at the first 20 reranked children and keeps those scoring at least 0.30:

selected_children = [
    child
    for child in reranked_children[:20]
    if child.score >= 0.30
]

That threshold is a filtering policy for this particular model and pipeline. It shouldn't be read as a universal probability of relevance, and switching rerankers would mean evaluating and calibrating it again. Tables need extra care because some exceed the reranker's 512-token input ceiling. Before tokenizing, the workflow flags table pairs that are too long, so they neither cause indexing errors or silently disappear.

Expanding children into parent context

The reranker still works on children because they make focused comparison units. Returning only those children, though, could drop surrounding clauses, conditions, exceptions or related articles. Each child therefore keeps the parent_id assigned during chunking:

{
  "child_id": "parent_0082_child_1",
  "parent_id": "parent_0082",
  "text": "23. Prohibition of traffic in human beings and forced labour..."
}

Once filtering by threshold is done, the system takes the parent_id from each selected child and looks up the whole parent in the parent-chunk collection:

parent_id = selected_child["parent_id"]
parent = parent_lookup[parent_id]

Several selected children can share one parent, and returning it repeatedly would duplicate context, so children are grouped by parent_id. Each unique parent is returned once with the children that led to it, and the top child score becomes the score of that parent source:

Selected child 1 ──┐
Selected child 2 ──┼── parent_0082
Selected child 3 ──┘

A small child containing the relevant language of Article 23 might pull in parent_0082, whose parent then supplies the broader "Right against Exploitation" section, including surrounding clauses and related Article 24 content. That keeps the split of duties intact: child chunks decide which evidence is relevant, and parent chunks provide the context in which it appears.

The returned parent source also keeps its document ID and heading path, the IDs of matched children, their block types and reranker scores, so a returned source can be traced back through the pipeline to understand why it was chosen. The hybrid retrieval experiment is in code/hybridrag.ipynb.

4. Query expansion and multi-hop questions: closing the vocabulary gap

The hybrid pipeline worked well when the user's wording closely echoed the Constitution. A direct question like "What is Article 21?" contains an exact reference, which gives BM25 and dense retrieval a strong signal.

Conceptual questions were harder. Users often describe a principle with terms from judgments, textbooks or legal commentary and not the Constitution's own language. A question about the "basic structure doctrine" may need Articles 13 and 368 even though the phrase basic structure appears in neither. In other words, the user has the legal idea right while the document names it differently.

What the evaluation showed

The problem was tested in a Kaggle notebook covering direct questions and a harder set of conceptual questions mapped to expected constitutional articles. The experiment predates the BGE migration. At that point the 871 children carried embeddings from nomic-ai/nomic-embed-text-v1.5, and queries were encoded with the same model and no retrieval prefixes. That detail matters because the experiment was meant to isolate what rewording the query does, and not to rank embedding models, so the same Nomic-based hybrid pipeline handled both versions of each conceptual question.

A question counted as passed only if the retrieved parent context contained every target article tied to it, so the reported percentage measures full-target retrieval coverage and not the factual accuracy of any LLM-written answer. The telling comparison is between the two conceptual sets. Query expansion lifted the retrieval pass rate from 56.0% to 74.5%, a gain of 18.5 percentage points. It recovered the full target set for 37 more questions and cut failures from 88 to 51.

The result suggested that many failures weren't caused by missing documents or bad chunk boundaries. The evidence existed in the corpus, but the original query didn't give retrieval the vocabulary needed to find it.

Expanding without changing intent

Expansion restates a conceptual question in words the source is more likely to contain. Suppose the question asks what restricts Parliament from altering the essential identity of the Constitution. The expanded form might bring in the basic structure doctrine, the core of the Constitution that cannot be amended, laws that conflict with Fundamental Rights, Article 13, and the amending power in Article 368. What the user wants hasn't changed, but the longer text adds retrieval signals such as:

• the common name of the doctrine,
• related constitutional terminology,
• alternative expressions of the same idea,
• relevant institutional or structural language, and
• explicit article numbers, when they can be inferred confidently.

The same treatment covered severability, pith and substance, judicial activism, and how Fundamental Rights relate to the Directive Principles.

In the Kaggle runs the expanded questions were written in advance and even listed probable articles, so it should be read as a controlled query-reformulation experiment and not an evaluation of an automatic expansion model. It shows how much retrieval can improve when the query contains source-aligned language, but a live query planner would need separate evaluation without access to target labels. The later dense benchmark swapped Nomic for BGE because BGE did better across the fixed child corpus, yet the Kaggle result still shaped the design: a better embedding model improved semantic retrieval, while query expansion tackled a different problem, the vocabulary gap between user and document.

Expansion versus subqueries

Expansion helps when one information need is phrased in unfamiliar or indirect terms, and it yields one richer query. Multi-hop questions are different, because they contain several information needs that may point to different parts of the document. Consider a request to compare the remedies under Articles 32 and 226. Searched as one long query, the signals compete: an excellent hit for Article 32 can crowd the candidate list while Article 226 gets too little representation, and reranking the combined candidates doesn't guarantee both sides survive.

A multi-hop question can instead be split into focused subqueries:

1. What remedies and jurisdiction does Article 32 provide?
2. What powers and jurisdiction does Article 226 provide?
3. How do Articles 32 and 226 differ constitutionally?

Each subquery runs through the whole retrieval process (BM25 and dense retrieval, then rank fusion, then cross-encoder reranking, then parent expansion), and the resulting parents are merged and deduplicated into combined evidence. This stops one part of a complex question from using up capacity meant for another, and it makes retrieval easier to inspect, since each branch matches a specific need.

The resulting routing strategy

The evaluation showed that no single query transformation suits every question. Direct questions already carry strong signals and should stay as they are. Conceptual questions benefit from vocabulary expansion, and multi-hop questions need splitting so each requirement gets its own branch:

• Direct factual or article-specific questions: search with the original query.
• Conceptual questions with a vocabulary mismatch: expand into one source-aligned query.
• Multi-hop, comparative or multi-article questions: break into focused subqueries and run each branch on its own.

This routing avoids altering simple queries needlessly while giving complex ones the structure they need. The original Kaggle evaluation is implemented in code/hybrid_eval.ipynb, and the later expansion and fan-out experiments appear in code/hybridrag.ipynb.

Lessons from the project

The main lesson is that better RAG rarely hinges on one perfect model. Retrieval quality comes from connected decisions about extraction, chunking, embeddings, candidate combination, reranking and context recovery. Evaluation changed several choices along the way. The dense benchmark replaced Nomic with BGE while keeping the semantic boundaries, the Kaggle experiment showed conceptual questions benefit from terminology closer to the source, and multi-hop failures pointed toward focused subqueries and independent retrieval branches.

Natural next steps include wrapping the pipeline in a chatbot application with a structured backend and a front end for asking questions and inspecting sources, then packaging it with Docker and deploying it to a cloud platform, where model loading, memory use, latency, persistent indexes, logging, scaling and cost all become questions. The system is still evolving and some failures remain unresolved, which is the point of the exercise: keep every stage inspectable, find where evidence gets lost and let evaluation guide the next improvement.