Splitting Markdown for RAG: Five Experiments With LangChain Headers

Chopping a Markdown file into fixed-size character windows is a quick way to wreck its structure. Headings end up in one chunk and their paragraphs in another, and a list gets sliced mid-item. If that text is feeding a RAG pipeline, retrieval gets worse for it. LangChain's MarkdownHeaderTextSplitter takes a different route: it cuts at headings and records which headings a piece of text sat under as metadata.

The easiest way to see what it does is to change one setting at a time on a tiny document and read the output. That's what follows.

What the splitter does

The class splits Markdown text wherever the headers you specify appear. Take this miniature document:

# Header 1

Here is the test content of Header 1

## Header 2

1. Content
2. Content
3. Content
4. Content

The # and ## lines give the text its shape. The splitter treats those lines as split points and places the heading text in each resulting document's metadata. For RAG that's handy, because every chunk keeps a record of where in the original file it came from.

The test file

Every experiment below uses the same file, simple_water_cycle.md, a deliberately small page about the water cycle. It has two heading levels. A top-level heading, "The Water Cycle", opens the page and is followed by one introductory sentence: the water cycle describes how water moves through the environment. Under a second-level heading, "Main Stages", sit four numbered stages (evaporation, condensation, precipitation and collection), and then a closing sentence saying the cycle repeats continuously. Stage names are in bold.

Setup

Start with pathlib and read the file through its Path object. Then define the list of headers to split on. Each entry is a pair: the Markdown symbol first, then the metadata key to store that heading under. Here the pairs are # mapped to "Header 1" and ## mapped to "Header 2". Time to experiment.

Experiment 1: the defaults

Run the splitter with no extra options. The result is two Document objects.

The first holds the intro sentence, with metadata of {'Header 1': 'The Water Cycle'}. The second holds the four stages plus the final sentence, and its metadata carries both levels: {'Header 1': 'The Water Cycle', 'Header 2': 'Main Stages'}.

Two things stand out. First, the heading lines themselves aren't inside page_content by default. The text # The Water Cycle never shows up in the first document's content, because that information lives in metadata instead. Second, the parent heading follows the child along. The "Main Stages" section's metadata includes "Header 1" even though that heading sits above it. So the splitter does more than cut text. It keeps the document's heading hierarchy alive through metadata.

Experiment 2: keeping headings in the text

Headings are stripped from content unless you say otherwise. Setting strip_headers=False changes that. Now the first document begins with # The Water Cycle followed by the intro sentence, and the second begins with ## Main Stages and then its list.

The splitting points and the metadata stay identical. Only the presence of the heading text inside page_content differs. By default a header lands in metadata alone; with this flag it lands in both metadata and content. That's what you want when the real heading words should travel with the text to whatever consumes it next, such as an embedding model or a prompt.

Experiment 3: return_each_line

Go back to the default heading behavior and flip only return_each_line=True. The output grows from two documents to three. The first is still the intro. The second is the four numbered stages. The third is the closing sentence about the cycle repeating. Documents two and three both carry the full "Header 1" plus "Header 2" metadata.

Don't read that as one document per physical line. The four list items stay together as a single block, while the final paragraph becomes its own document. What you get is finer-grained content blocks, each with the right header metadata attached.

Experiment 4: both options together

Now combine strip_headers=False with return_each_line=True. Five documents come back, in this order: the top heading, the intro paragraph, the second-level heading, the stage list, and the final paragraph. The headings turn into standalone documents, with page_content='# The Water Cycle' and page_content='## Main Stages', and metadata is still attached.

The two flags do separate jobs. One decides whether headings remain in the content. The other decides how fine the content blocks are. Together, headings become documents of their own.

How hierarchy gets inherited

The header list used throughout was # and ##, and the document nests like this: "The Water Cycle" at the top, "Main Stages" beneath it. Metadata mirrors that nesting. Text directly under the top heading gets only {'Header 1': 'The Water Cycle'}. Text under "Main Stages" gets both keys. In short, a child section inherits its parent heading's metadata, one of the most useful properties of splitting this way.

Experiment 5: add a second splitter

A section produced by heading-based splitting can still be too big for a RAG system. The fix is to feed these Documents into RecursiveCharacterTextSplitter, which cuts them into smaller pieces. Each smaller piece still knows which section it belongs to, and that metadata helps later when you need to work out the context of a retrieved chunk.

On the sample file, the intro paragraph stays a single Document. The four numbered stages, which had been one block, now become four separate Documents, one each for evaporation, condensation, precipitation and collection. Every one of them keeps the "Header 1" and "Header 2" metadata. The order matters: the Markdown splitter first uses the document's structure, then the recursive splitter shrinks the results.

Takeaways

Using one small file, you've seen that MarkdownHeaderTextSplitter splits on headings and stores the heading hierarchy as metadata. strip_headers controls whether heading text stays in the content, and return_each_line produces finer Documents. Pairing it with RecursiveCharacterTextSplitter yields smaller chunks without losing the section information. For structured Markdown headed into RAG, that combination gets you both a workable chunk size and the context around each piece.