LangChain Document Loaders: Getting Files and Pages Into a RAG Pipeline

An LLM can't wander through your folders and absorb the files on its own. Retrieval-augmented generation exists because models don't know your private data, miss anything recent and sometimes make things up when context is missing. The fix is to fetch the right text at query time. But before any of that happens, something has to get a PDF, a web page, a CSV or a Word file into the pipeline at all. That job belongs to the document loader, which is easy to overlook and worth getting right.

What a loader does

A loader reads data from some source and turns it into a format LangChain understands. It acts as a translator. Your material might sit in any of these places:

• PDFs
• Plain text files
• CSV files
• Websites
• Word documents
• Databases

Everything after the loader expects one standard shape, and the loader's role is to close that gap.

Why PDFs can't go straight to chunking

Beginners often picture the flow like this:

PDF → Chunking → Embeddings

Real life is messier. A PDF isn't text. Inside it you'll find fonts, images, layout data, tables, headers and footers, and metadata. A chunker only works on text, so something has to pull the words out first. The actual flow is closer to:

PDF → Document Loader → Text Extraction → Chunking → Embeddings

The loader extracts the text and packages it in a shared format.

Loaders you'll meet most

LangChain ships loaders for many sources. The input changes but the principle doesn't: read from somewhere and produce LangChain Documents. These are the ones you'll reach for most often:

• TextLoader handles plain .txt files.
• PyPDFLoader handles PDFs.
• WebBaseLoader handles web pages and URLs.
• CSVLoader handles CSV files.
• DirectoryLoader handles many files in a folder.

One output type: Document

Whichever loader runs, you mostly get back a list of Document objects. Each carries a text body and some details about where it came from:

Document(
    page_content="Employees are eligible for 24 paid leaves per year.",
    metadata={
        "source": "employee_policy.pdf",
        "page": 5
    }
)

Two fields matter most. page_content is the text itself, and metadata describes the source, here a file name and page number. This standard shape is handy because later stages don't care where the data originated. A PDF loader and a web loader both give you Documents, and the chunking step treats them identically.

TextLoader

The simplest option reads plain text files. Import it first:

from langchain_community.document_loaders import TextLoader

Then point it at a file, load it and print the result:

loader = TextLoader("notes.txt")documents = loader.load()print(documents)

You get a list back:

[
    Document(...)
]

Even for a single file you get a list, which keeps the interface consistent. Each element is a Document.

Use it when your data is already text, such as logs, notes, text exports or clean datasets. It's very basic, though. It can't parse PDFs, read tables, understand layout or extract images. If the source isn't already text, pick something else.

PyPDFLoader

PDFs are probably the most frequent source in RAG projects, and PyPDFLoader is the usual starting point. Import it:

from langchain_community.document_loaders import PyPDFLoader

Then load a report:

loader = PyPDFLoader("annual_report.pdf")documents = loader.load()

Suppose that file has 25 pages. Counting the result:

len(documents)

gives you:

25

Each page becomes its own Document. Looking at the first one:

documents[0]

shows this:

Document(
    page_content="Page 1 text...",
    metadata={
        "source": "annual_report.pdf",
        "page": 0
    }
)

So a 25-page PDF yields 25 Documents. Under the hood this loader relies on the PyPDF library. It does well on simple PDFs, but it struggles with complex layouts, multi-column pages, tables and scanned files, which is why the choice of PDF loader matters.

Picking a PDF loader

No single loader is best for every file. A rough map:

• PyPDFLoader suits simple, text-based PDFs.
• PDFPlumberLoader fits PDFs with tables and structured layouts.
• PyMuPDFLoader is quick and a solid general-purpose pick.
• UnstructuredPDFLoader targets complex enterprise documents.
• AmazonTextractPDFLoader covers scanned files and OCR-heavy work.

A workable rule of thumb: start with PyPDF. If the extracted text looks bad, move to PyMuPDF or Unstructured. If the document is a scan, use an OCR service such as Textract.

DirectoryLoader

Loading one file is trivial. Loading 5,000 PDFs or 10,000 text files by hand is not, and that's what DirectoryLoader is for. Import it:

from langchain_community.document_loaders import DirectoryLoader

Then give it a folder and a glob pattern:

loader = DirectoryLoader(
    "data/",
    glob="**/*.pdf"
)documents = loader.load()

That picks up every PDF under the directory. Imagine a layout like this:

data/
├── report1.pdf
├── report2.pdf
├── report3.pdf

The loader walks through each matching file and loads it, giving you:

[
    Document(...),
    Document(...),
    ...
]

For a big folder that could mean hundreds or thousands of Documents.

The problem with huge folders

Picture 20,000 PDFs, or 100,000 text files, and this call:

documents = loader.load()

LangChain reads everything into memory right away. That's eager loading. It's easy to use and fine for small datasets, but it costs a lot of memory and slows startup.

Eager versus lazy

There's an alternative:

loader.lazy_load()

With the eager version, everything loads at once:

documents = loader.load()

and the return type is:

List[Document]

The lazy version only produces documents as they're needed:

documents = loader.lazy_load()

and returns a generator:

Generator[Document]

Think of streaming video. A service doesn't download the full film before you press play, it feeds you data on demand. Lazy loading works the same way.

Use load() for small or medium collections and for prototyping. Use lazy_load() for large collections and memory-constrained environments.

WebBaseLoader

Not all knowledge lives in files. Sometimes it's on a website, and WebBaseLoader handles that. Import it:

from langchain_community.document_loaders import WebBaseLoader

Then pass a URL:

loader = WebBaseLoader(
    "https://python.langchain.com"
)documents = loader.load()

Internally it uses BeautifulSoup to parse the HTML and pull out text, which suits blogs, documentation, articles, knowledge bases and company sites.

It works best on static pages. JavaScript-heavy sites, dynamic content and client-side rendering give it trouble. For those, SeleniumURLLoader can render the page in a browser before extracting anything.

CSVLoader

Structured data often arrives as CSV. Take a small employee file:

employee_id,name,department
1,John,Engineering
2,Sarah,HR

Import the loader:

from langchain_community.document_loaders import CSVLoader

And load the file:

loader = CSVLoader(
    file_path="employees.csv"
)documents = loader.load()

Each row turns into a Document, with the column names and values written out as text and the row number kept in metadata:

Document(
    page_content="""
employee_id: 1
name: John
department: Engineering
""",
    metadata={
        "source": "employees.csv",
        "row": 0
    }
)

It fits product catalogs, customer records, inventory, financial records and similar business data.

Three things to keep

1. Every RAG pipeline begins with loading. Before chunking, embeddings or retrieval, the data has to get in.
2. Different sources need different loaders: text files use TextLoader, PDFs use PyPDFLoader, sites use WebBaseLoader, CSVs use CSVLoader, and folders use DirectoryLoader.
3. Nearly everything ends up as a Document object, and that common shape lets the rest of the pipeline work consistently.

The full catalog of loaders is in the LangChain integrations docs.

Once the documents are loaded, the next puzzle is how to split them: by characters, words, sentences, paragraphs or tokens, and at what size for a 100-page file. Cut too small and you lose context. Cut too large and you retrieve noise. That decision deserves its own careful treatment, because it's the art of slicing documents without breaking your RAG system.