Docling Pipelines: Building Enterprise-Grade RAG Data Pipelines

Where glue code goes wrong

Imagine a shared drive holding 10,000 PDFs, a SharePoint site stuffed with Word files, and a backlog of recorded customer calls. A product team upstream wants every bit of it in a vector store, searchable and chunked, with personal data scrubbed and duplicates gone.

The usual engineering response is a script, then a second script. A few libraries get pulled in, a Pandas notebook ties them together, a cron job reruns the lot every night, and it ships. It works, for a while.

Three months on, someone asks why new documents are missing. The trail ends at a schema change in a downstream column, and the script had quietly discarded those rows. Nobody saw it, because nothing was observable. After the fix comes the rerun, and with it the realisation that there is no record of which documents were already handled, so all 10,000 get processed again.

That is the typical life of document-pipeline glue. It begins simple and decays slowly, then abruptly. Retries get bolted on, quality filtering arrives as an afterthought, and dependency tracking between steps doesn't exist, so success depends on scripts happening to run in the right order. Incremental updates are painful without a stable document identity. What remains is a tangle that only its author can debug, and even the author is no longer sure.

What docling-pipelines is

Docling-pipelines is a modular, operator-based framework for document processing, meant for building complete RAG data pipelines. Its scope is the full lifecycle, starting with ingestion from local or cloud sources, then text and entity extraction, quality filtering, chunking and embedding, and ending with a write to a vector database. That is the standard path, and most production pipelines are a variation on it.

It installs as an ordinary Python package:

bash
pip install docling-pipelines

It runs anywhere Python does. For basic use it needs no external infrastructure: no Kafka, no Spark cluster, nothing to set up. A pipeline is described in a JSON file, the CLI is pointed at that file, and the run starts.

There are three ways in. The CLI suits one-off runs and local development. The Python library, centred on DocpipeFlowManager, fits triggering pipelines from application code or notebooks. The REST API is meant for pipelines run as a shared service that several teams use. All three get covered below.

The project is open source, and its first release, v0.1.0, came out in July 2025.

The core idea: operators and flows

Anyone new to the framework should spend the most time here, since the rest follows once this is clear.

The operator is the basic unit. An operator is a Python class that accepts a table of documents and returns a table, possibly transformed. Every capability in the framework, whether ingestion, extraction, language detection, chunking or embedding, is an operator.

The obligations are few:

• short_name: the string flow JSON uses to refer to the operator
• transform(table): takes a pa.Table and returns a list of tables plus a metadata dict
• get_metadata(): a @staticmethod returning operator metadata so it can be discovered
• get_required_features(): states which columns the operator needs on input
• validate(): an optional hook that checks configuration at startup

A minimal skeleton looks like this:

from docpipe.core.operators.abstract_operator import AbstractOperator, OperatorCategory
import pyarrow as pa

class MyOperator(AbstractOperator):
  short_name = "my_operator"
  category = OperatorCategory.Quality

  def __init__(self, config: dict) -> None:
    super().__init__(config)

  def transform(self, table: pa.Table) -> tuple[list[pa.Table], dict]:
    metadata = self.create_base_metadata(total_docs_count=len(table))
    # … process table …
    return [table], metadata

  @staticmethod
  def get_metadata() -> dict:
  return {"short_name": "my_operator", "description": "…"}

Implementing those methods yields a first-class component that composes with every other operator in the ecosystem.

PyArrow tables carry the data. Each operator is given a pa.Table and returns one. That is a deliberate choice, because PyArrow's columnar format is efficient and typed, and it moves between stages without copying. Raw strings and Python dicts never travel between steps.

The neat consequence is that columns pile up as data advances. Ingest adds id, name, path and source. Extraction adds content. The chunker adds chunked_content, and the embeddings step adds embeddings. By the time a row arrives at the vector store it holds a full record, with all its columns and metadata, ready to index.

An operator only touches the columns it cares about, and everything else flows past untouched. A new operator can therefore be dropped in anywhere without changing its neighbours.

A flow is a DAG. The pipeline lives in a JSON file. Each node declares a type (the operator's short name), a name that is unique in the graph, a config block and a depends_on list. At startup the engine resolves the dependency graph and runs nodes in topological order, with parallel branches executing at the same time.

So the pipeline's structure is data and not code. It can be versioned, diffed, statically validated and replaced without touching any Python. The --validate option flags dependency violations, missing columns and config errors before a single document is handled.

Why the model works. Two benefits stand out in practice.

One is composability. An operator can follow any other provided the upstream ones have produced the columns it lists in get_required_features(). The engine checks this at startup and reports the exact missing column and the node that should have supplied it, all before the run begins.

The other is testability. Each operator behaves as a pure function from table to table, so it can be tested alone: build a pa.Table with the needed columns, call transform() and assert on the result. No mocks, no side effects and no live pipeline are required.

A tour of the operators

There are 22 built-in operators in five categories. This is a quick survey.

Ingest. IngestSource is the one ingest operator, covering local and cloud sources alike. A local path uses the filesystem provider, and the same node can switch to any of eight cloud providers (S3, IBM COS, SharePoint, OneDrive, Google Drive, Box, CSV files and web pages) with nothing else in the flow changing. It outputs a row per file with metadata only and no content yet. The extract step fetches content later, which keeps ingestion quick and light on memory even at scale.

Extract. Extract turns raw files into structured text. PDF, DOCX, HTML, images, audio and video all become markdown through Docling, either in library mode (local, no network call) or through Docling Serve in API mode, which suits offloading heavy extraction to a dedicated service.

Entity extraction is optional on top of that, with three providers: LiteLLM (which routes to over 100 model providers), IBM watsonx.ai, and Docling's template-based extraction. The choice depends on model access and cost.

ACLOperator is part of the extraction stage. It pulls access-control metadata out of documents into a column of its own, so systems further down can apply document-level security with no second trip to the source file.

Functional. This category holds the pipeline logic: operators that transform, route and reshape data instead of enriching it.

ChunkerOperator cuts extracted text into chunks using one of three strategies: simple (fixed character size, configurable overlap), semantic (content-aware) or hybrid (Docling-native chunking that keeps document structure). It can also summarise each chunk with an LLM, which gives retrieval a richer representation.

EmbeddingsOperator creates vector embeddings for every chunk. It works with HuggingFace models run locally, LiteLLM (100+ cloud providers) and IBM watsonx.ai, and swapping the embedding model is a one-line change in the flow JSON.

BranchingOperator is where flows get interesting. It splits documents into separate sub-pipelines by column values, SQL-style conditions or classification labels. Each branch runs on its own and is referenced by name in depends_on with dot notation, for example "depends_on": ["branch_node.branch_name"].

MergeOperator joins branched streams back together, either stacking rows or joining columns, according to what the downstream operators expect.

DocIdHashOperator derives a deterministic, stable hash ID from each document's content. That ID underpins deduplication and incremental processing: the same document always gets the same ID, wherever and whenever it was ingested.

EntityCurationOperator and NOOPOperator complete the set. The first handles cleanup of entities after extraction, and the second is a passthrough node that helps with wiring the DAG.

Quality. Operators in this group mark the gap between merely having a pipeline and having one that can be trusted.

LanguageDetect identifies a document's language with fastText and adds a lang_name column that can drive filtering, routing or tagging.

ReadabilityOperator scores each document five ways for readability: Flesch reading ease, the Flesch-Kincaid grade, Gunning Fog, Dale-Chall and a tally of difficult words. They make useful routing signals, for example to tell technical documentation from general content before choosing a chunking strategy.

PIIAndHAPAnnotator uses LLM-based detection to find and redact personal information (email addresses, phone numbers, SSNs, card numbers, bank accounts, IP addresses) along with hate, abuse and profanity, with thresholds set per category.

RedactionOperator covers pattern-based redaction where formats are known, such as internal IDs, product codes or anything a regex can match.

EdedupOperator removes exact duplicates across the document set, using the doc_id_hash column to do so deterministically.

SQLFilterOperator is among the most underrated pieces. It filters rows using a SQL WHERE condition on column values, so "docq_total_words > 100 AND lang_name = 'en'" forms a complete quality gate without any Python. Together with readability scores and language detection, it allows fairly elaborate routing with no code.

DocumentClassifierOperator sorts documents into configurable categories. DocQuality computes quality metrics for the whole corpus, and MLEnrichmentOperator adds ML predictions as extra columns. The wider point, which other pipeline write-ups also make, is that data quality often moves RAG results more than the choice of model does.

VectorDB and storage. VectorDBOperator writes chunked, embedded documents to a vector store, with OpenSearch and Milvus as supported backends. Field mappings, index settings and metadata normalisation are all set in the flow JSON.

DocumentSetOperator saves the processed table into a DuckDB-backed document set. That helps when processed documents should be queryable without a vector store, or when a durable mid-pipeline checkpoint is wanted.

StorageOutputOperator writes documents back to external destinations (S3, SharePoint, Google Drive or the local filesystem), in processed or original form, which suits "process and archive" patterns.

All of these compose in any order as long as upstream dependencies are met, so the pipeline isn't locked into a fixed shape.

A real flow

Here is the canonical quickstart flow, a complete ingest-to-vector-store pipeline that uses Ollama for embeddings and OpenSearch for storage:

{
  "flow_name": "complete-document-pipeline",
  "global_config": {
    "doc_column": "content",
    "storage": "in-memory"
  },
  "flow": [
    {
      "type": "ingest_source",
      "name": "ingest_local_folder",
      "config": {
        "provider": "filesystem",
        "connection_params": {
          "paths": [
            "./sample_documents"
          ]
        },
        "include_filter": "pdf,txt,docx"
      }
    },
    {
      "type": "extract_operator",
      "name": "extract_with_docling",
      "config": {
        "text_extraction": {
          "provider": "docling_library"
        },
        "entity_extraction": {
          "provider": "none"
        }
      },
      "depends_on": [
        "ingest_local_folder"
      ]
    },
    {
      "type": "chunker",
      "name": "simple_chunker",
      "config": {
        "chunk_type": "simple",
        "chunk_size": 512,
        "chunk_overlap": 50
      },
      "depends_on": [
        "extract_with_docling"
      ]
    },
    {
      "type": "embeddings",
      "name": "ollama_embeddings",
      "config": {
        "provider": "litellm",
        "provider_config": {
          "model_id": "openai/nomic-embed-text",
          "api_base": "http://localhost:11434/v1"
        },
        "embeddings_column": "embeddings"
      },
      "depends_on": [
        "simple_chunker"
      ]
    },
    {
      "type": "vectordb",
      "name": "opensearch_vector_store",
      "config": {
        "provider": "opensearch",
        "doc_id_column": "doc_id_hash",
        "embeddings_column": "embeddings",
        "provider_config": {
          "index_name": "sample-documents-index",
          "host": "localhost",
          "port": 9200
        }
      },
      "depends_on": [
        "ollama_embeddings"
      ]
    }
  ]
}

Going node by node:

• ingest_local_folder scans ./sample_documents for PDF, text and DOCX files by way of ingest_source and its filesystem provider. Each file becomes one row carrying id, name, path and source. Contents aren't read at this point, so the step is quick: it finds files instead of loading them.
• extract_with_docling takes those metadata rows, reads each file with Docling and adds a content column holding extracted markdown. This is the point where PDFs turn into structured, queryable text. The docling_library provider runs wholly in-process, so no external service is needed.
• simple_chunker cuts the content column into 512-character chunks with 50 characters of overlap between neighbours. It adds chunked_content, a structured list of chunk objects that each carry their text and position in the source document.
• ollama_embeddings sends every chunk to the nomic-embed-text model served locally and stores the resulting vectors in a new embeddings column. It is simply LiteLLM aimed at Ollama, so changing model_id and api_base routes it to any of the 100+ providers LiteLLM supports, with nothing else altered.
• opensearch_vector_store upserts the chunked, embedded rows into an OpenSearch index and uses doc_id_hash as the primary key. Rerunning it therefore never creates duplicates.

A few structural points deserve attention. depends_on is the only wiring required: there is no explicit data passing, no callback registration and no shared state, because the engine resolves the graph and moves the data. The global_config block sets defaults that every operator inherits. doc_column says which column holds the main text, and storage: "in-memory" keeps intermediate tables in RAM between steps, which suits most runs and can be swapped out for large datasets.

The whole pipeline runs with one command:

docling-pipelines - flow-file pipeline.json

Advanced patterns

Flows needn't be linear chains. Three patterns recur in real deployments.

Branching and merging. Branching sends documents to different sub-pipelines according to configurable criteria. Each branch is a named output that downstream nodes reference with dot notation in depends_on. A routing branch driven by readability score looks like this:

{
  "name": "quality_branching",
  "type": "branching",
  "config": {
    "branches": {
      "high_quality_branch": {
        "link_name": "High Quality Documents",
        "criteria_json": {
          "criteria_list": [
            {
              "variable": "flesch_reading_ease",
              "operator": ">",
              "value": 60
            }
          ],
          "logical_operator": "AND"
        }
      },
      "low_quality_branch": {
        "link_name": "Low Quality Documents",
        "criteria_json": {
          "criteria_list": [
            {
              "variable": "flesch_reading_ease",
              "operator": "<=",
              "value": 60
            }
          ],
          "logical_operator": "AND"
        }
      }
    }
  },
  "depends_on": [
    "readability"
  ]
}

The high-quality branch could go on to ML enrichment before embedding, while the low-quality branch passes through another SQLFilterOperator that drops documents with zero words before embedding. Both end up in separate OpenSearch indices, and MergeOperator can recombine the streams afterwards when later logic wants a single view.

Quality-gated routing. The idea is to run cheap quality operators first and then let SQLFilterOperator discard rows before any costly LLM step. A full gate may look like this:

lang_detect → ededup → doc_quality → readability → sql_filter → embeddings

The sql_filter condition is simply a SQL WHERE clause expressed as data:

{
  "criteria_list": [
    {
      "variable": "docq_total_words",
      "operator": ">",
      "value": 100
    },
    {
      "variable": "lang_name",
      "operator": "=",
      "value": "en"
    }
  ],
  "logical_operator": "AND"
}

There is no Python and no custom operator, only a condition in JSON, and every row that fails it never reaches the embedding call.

Multi-stage enrichment. The sequence classify, route, extract, enrich, store turns up constantly in production. Documents might be classified first, with technical documentation routed to an entity extractor tuned for technical content and policy documents routed to a different one. Each branch adds enrichment columns specific to its type, and the streams merge ahead of chunking and embedding so the remainder of the pipeline is shared.

Every stage adds columns, and downstream operators see an increasingly rich table. None of this relies on special modes or flags. These are ordinary operators composed together, which is the payoff of the operator model.

Three ways to run it

The framework is more than a CLI tool and fits several engineering contexts.

CLI. This is the quickest route to a running pipeline:

docling-pipelines - flow-file pipeline.json
docling-pipelines - flow-file pipeline.json - validate # validate without running
docling-pipelines - list-operators - verbose # discover all registered operators

--validate earns its keep while a flow is being developed: dependency violations, absent required columns and config type errors all surface before anything is read or written. Running it in CI ahead of deploying a flow change is a good habit.

Python library. When pipelines need triggering from application code:

from docpipe.lib.docpipe_flow_manager import DocpipeFlowManager
manager = DocpipeFlowManager(flow_file="pipeline.json")
manager.execute()

This fits code embedded in a bigger application, a notebook, or an orchestration script that needs to steer the run lifecycle itself.

REST API. For running as a long-lived service:

uvicorn docpipe.api.main:app - host 0.0.0.0 - port 8000

The FastAPI server exposes three groups of routes: /api/v1/flows to manage flows, /api/v1/job_runs to control job runs, and /api/v1/operators to discover operators. Users authenticate via LDAP or OAuth2/OIDC using JWT tokens, and OpenAPI/Swagger documentation is generated automatically at /docs. This is the interface for a shared service that several teams use, external events trigger, and others query for status and results.

Extending with custom operators

For lasting adoption inside real organisations, this is arguably the most important part of the design.

The contract is compact enough that an operator takes roughly 30 lines. A Python class qualifies if it inherits from AbstractOperator and implements transform(), get_metadata() and get_required_features(). It combines with every built-in operator, appears in the --list-operators output, and flow JSON refers to it by short_name just like a built-in.

Registration works in three ways, matched to the stage of development.

Filesystem path is for development. Point an environment variable at a local directory:

DOCPIPE_CUSTOM_OPERATORS=./my_operators docling-pipelines - flow-file pipeline.json

Everything in that directory is imported at startup. No installation is needed, so iteration is fast.

Python package is for distribution. Publish operators as a pip package with the right entry point, install it in the same environment, and they register themselves. This is how operators reach other teams without anyone touching the core codebase.

S3 URI is the enterprise option. Point DOCPIPE_CUSTOM_OPERATORS at a remote operator package and the framework fetches and registers it at runtime. It fits centralised operator management across many deployments: one place to update, and every deployment picks the change up on its next start.

Enterprise-specific logic therefore stays out of the core. Compliance processors, domain-specific entity extractors and proprietary scoring models can all be packaged as operators, composed with built-ins in flow JSON and deployed without forking the framework.

Operational concerns

A fair question about any tool like this is whether it is a good prototype or genuinely production-grade. The specifics follow.

Observability. Native OpenTelemetry support is built in, and every operator execution produces a span. Logs are structured JSON with trace correlation, carrying trace_id and span_id on every line. This works with Jaeger, Grafana Tempo, Datadog and AWS X-Ray with no configuration on the application side, and the overhead is zero when telemetry is switched off.

Incremental processing. The pipeline uses the deterministic ID from DocIdHashOperator to track what has been processed. On a rerun only new or changed documents are handled, and the same document submitted twice gets the same hash and is skipped. Setting force_ingest: true in global_config overrides that and reprocesses everything. Metadata for incremental runs can live in DuckDB (the default), PostgreSQL or the filesystem.

Security. ACLOperator surfaces permission metadata as a column for downstream enforcement, and PIIAndHAPAnnotator finds and redacts sensitive content before it reaches the vector store. Secrets in flow configuration can come from HashiCorp Vault: a path such as vault://path/to/secret written in the JSON gets resolved by the engine while the run executes. Access to the REST API is guarded with LDAP and OAuth2/OIDC.

Scale. Execution comes in three modes that stretch from a laptop to a distributed fleet. The default is an in-process thread pool, which is fast, simple and dependency-free. A process pool runs operators in separate processes for CPU-bound work on one node. For distributed production workloads there is a Prefect work pool running Docker containers, which adds horizontal scaling and central management of runs. Micro-batching settings control memory use in every mode, which matters when large documents go through LLM extraction.

Where to go from here

The project is hosted on GitHub under the IBM organisation, and its quickstart guide is the fastest way to a running pipeline. Areas that deserve closer study include the extraction layer (Docling integration, VLM pipelines for image-heavy documents, ASR for audio and video), writing a custom operator with non-trivial logic from the first class declaration through testing, registration and deployment, running Prefect with Docker for high-volume workloads, and the field mapping and schema template options of VectorDBOperator for OpenSearch.