How docling-pipelines Stores Document Vectors for RAG With OpenSearch or Milvus
The vector database layer of IBM docling-pipelines for RAG: pluggable OpenSearch and Milvus adapters, stale chunk cleanup, auto dimensions, JSON flows.
This article is published in English.
Sources: The Indexing Layer: How docling-pipelines Prepares Your Documents for RAG
Any modern application that answers questions about private documents leans on one pattern, Retrieval-Augmented Generation, or RAG. Say someone asks which renewal terms apply to last year's contracts, or what caps on liability appear in supplier agreements. Before a language model can respond, something has to find the right passages among thousands of files and pass them along in a usable form. That something is a vector database, and the vectordb operator in IBM's open-source docling-pipelines project fills that role.
Below is the reasoning behind vector databases, how RAG depends on them, and how docling-pipelines turns the storage half of a RAG pipeline into something declarative: configured through a JSON flow file and extendable with custom operators when needed.
RAG and where the vector database sits
With RAG, a language model doesn't answer from memory alone. The system first retrieves the most relevant passages from a document collection and puts them into the prompt as context. The model then writes an answer anchored in those passages and not in statistical habits from training.
A vector database makes the retrieval step possible. In practice it works like this.
At index time, every chunk of the collection is converted into a numeric vector, its embedding, by a model that has learned to place text with similar meaning close together in a high-dimensional space. The vectors are stored next to the original text.
At query time, the user's question is embedded with the same model. The database runs a nearest-neighbour search and returns the chunks whose vectors lie closest to the question vector. Those chunks are relevant in meaning even when they share no keywords with the question. The model receives them as context and can produce a grounded answer.
Why an ordinary database can't do it
A relational or document database can match keywords but not meaning. A search for "termination clauses" won't return a paragraph about "contract exit conditions", even though they say the same thing. A vector database works on meaning and not on characters.
At scale, comparing a query against millions of vectors exhaustively would be too slow. Vector databases therefore use approximate nearest-neighbour indexes, such as HNSW (Hierarchical Navigable Small World graphs) and IVF (Inverted File Index). They give up a little recall for a large gain in latency, and both index types are ordinary configuration options in the vectordb operator.
Which backends docling-pipelines supports
The project ships with built-in support for two vector database backends, OpenSearch and Milvus. Both are configured through the flow JSON with no code changes. One "provider" key in the operator's config chooses the backend, and everything else follows from that.
The operator's hexagonal architecture makes other backends easy to add. To support Qdrant, pgvector, Weaviate or anything else, you write one adapter class and register it. Neither the operator nor the pipeline framework needs to change.
How the operator works
Within the indexing pipeline, the vectordb operator handles storage. Rows arriving in its PyArrow table are already enriched with these fields:
- the original document text;
doc_id_hash, which gives each document a fixed identifier that is the same on every run;- a
chunked_contentcolumn, which is the document split into overlapping windows; - an
embeddingscolumn with one numeric vector per chunk, produced by theembeddingsoperator upstream.
The operator takes that table and writes every chunk, with its embedding and metadata, into the vector database. Along the way it creates or validates the index schema, batches bulk inserts, tracks which chunks succeeded or failed, and removes stale chunks from earlier runs when a document has changed.
Hexagonal design: plug in any store
The operator is intentionally not tied to a single database. It hands the backend work to an adapter that fulfils a VectorStorePort interface, an abstract contract spelling out what each vector store has to offer, including create_index(), index_documents(), refresh_index() and delete_documents_by_ids().
This is the hexagonal architecture pattern applied to a pipeline operator. The logic about what to store lives in the operator. The infrastructure question of how to talk to a particular backend lives in the adapter. The operator only calls the port interface and never touches OpenSearch or Milvus protocols directly.
VectorDBOperator
│
└── VectorStorePort (interface / abstract contract)
├── OpenSearchAdapter (ships with docling-pipelines)
├── MilvusAdapter (ships with docling-pipelines)
└── YourCustomAdapter (implement VectorStorePort → plug in)
To support a database that isn't covered yet, implement VectorStorePort and register the adapter with the factory. After that, your adapter's name can be used as the "provider" in a flow JSON. Nothing else changes, whether that is the operator, the surrounding pipeline or existing flows with type: vectordb.
Chunked and unchunked documents
Both modes are handled transparently. When chunked_content exists and is filled, which is the output of the upstream chunker operator, each chunk is indexed as its own document. Its primary key is derived from the parent file ID, the chunk index and the chunk text. When the table holds whole documents with no chunking, each row is indexed as one unit. The operator picks the path automatically, and no flag is needed.
Cleaning up stale chunks
Updates are a familiar headache with chunked indexes. If a file is ingested a second time after its content was edited, the new chunks no longer match the ones stored earlier. The operator copes by looking up the existing chunk IDs for each document ahead of the insert, computing the difference between old and new sets and deleting the stale ones before writing the fresh batch. The index then reflects the current document however many times the pipeline has run over the same corpus.
Vector dimensions detected for you
You don't have to count dimensions and hard-code them. When the pipeline runs, the operator reads the first non-null embedding in the table and takes the dimension from its length. The index is then created with that size, or checked against it if it already exists.
A complete RAG indexing flow
A full indexing pipeline here is five operators linked by depends_on edges. This local example uses OpenSearch:
{
"flow_name": "rag-indexing-pipeline",
"global_config": {
"doc_column": "content",
"storage": "in-memory",
"execute_type": "local",
"enable_micro_batching": true,
"micro_batch_size": 10
},
"flow": [
{
"name": "ingest",
"type": "ingest_source",
"config": {
"provider": "filesystem",
"connection_params": { "paths": ["./documents"], "recursive": true },
"include_filter": "pdf,docx,txt"
}
},
{
"name": "extract",
"type": "extract_operator",
"depends_on": ["ingest"],
"config": {
"text_extraction": { "provider": "docling_library", "doc_column": "content" }
}
},
{
"name": "chunk",
"type": "chunker",
"depends_on": ["extract"],
"config": {
"doc_column": "content",
"chunk_size": 512,
"chunk_overlap": 50
}
},
{
"name": "embed",
"type": "embeddings",
"depends_on": ["chunk"],
"config": {
"provider": "litellm",
"embeddings_column": "embeddings",
"provider_config": {
"model_id": "openai/nomic-embed-text",
"api_base": "http://localhost:11434/v1",
"api_key": "${OLLAMA_API_KEY}"
}
}
},
{
"name": "store",
"type": "vectordb",
"depends_on": ["embed"],
"config": {
"provider": "opensearch",
"doc_id_column": "doc_id_hash",
"create_index": true,
"provider_config": {
"index_name": "my_rag_index",
"host": "localhost",
"port": 9200,
"username": "${OPENSEARCH_USERNAME}",
"password": "${OPENSEARCH_PASSWORD}",
"use_ssl": false,
"engine": "faiss",
"algorithm": "hnsw",
"space_type": "l2"
}
}
}
]
}
Run it from the command line:
docling-pipelines --flow-file rag-pipeline.json
Every PDF, DOCX and plain text file under ./documents is discovered, extracted and chunked into windows of 512 tokens with 50 tokens of overlap, settings that live in the chunker node and can be changed freely. The chunks are embedded with the nomic-embed-text model running locally through Ollama and finally written to an OpenSearch index that is set up for similarity search.
The example reads from a local folder, but ingestion also covers S3, SharePoint, Google Drive, Box, OneDrive and custom loaders. Only the ingest_source node changes. Every later operator, vectordb included, stays the same wherever the documents come from.
Milvus with two embedding models
Teams who want to embed documents with two different models at once, say a general-purpose model and a domain-specific one, can use the branching and merge operators. The flow runs ingest, extract, chunk, branch into two embedding steps, merge and then write to Milvus.
The Milvus adapter accepts several vector columns. Each model's output goes into its own vector field in the collection, which allows ensemble retrieval at query time: take the top results from each model's index and combine them before ranking.
Incremental indexing
The operator is built to run again and again over a growing corpus without re-indexing everything. The ingest operator also remembers earlier runs and skips any file whose hash and modification time are unchanged. Together, only new and modified documents travel through the pipeline. Their chunks are written, stale chunks of updated documents are removed and everything else is untouched.
For a scheduled pipeline over a continuously growing document store, run time scales with the amount of change and not with the total size of the corpus.
Where it fits in the bigger picture
In a docling-pipelines RAG flow, the vectordb operator is the vector storage node. Depending on the design it may be the last node, or it may be followed by others. A document_set operator, for instance, can persist the full enriched PyArrow table to DuckDB for later analysis alongside the vector index.
Every operator communicates through a standard PyArrow table schema, so vectordb is indifferent to where its rows originated, be it a local folder, S3, SharePoint or a custom loader. Portability of sources and of storage is a structural property enforced by the architecture and not left to convention. The operator covers index creation, batched bulk writes, multiple vector columns, stale chunk cleanup and automatic dimension detection, all driven by the flow file, with chunk size, overlap, embedding model, index algorithm and similarity metric exposed as configuration.
It is one piece of a growing pipeline ecosystem, and the project is open source under IBM. If your vector database isn't supported yet, the adapter interface is the place to start.