Driving Docling Pipelines Over HTTP: From Project Setup to Indexed Chunks
Walk through the Docling Pipelines REST API step by step: start the server, discover operators, validate and run an ingestion DAG, and read its execution telemetry.
This article is published in English.
Sources: **A Hands-On Guide to the Docling Pipelines REST API**
Turning a folder of PDFs and Word files into searchable vector chunks usually takes glue code for extraction, chunking, embedding and indexing. Docling Pipelines packages those stages as operators behind a FastAPI-based REST server, so any client, such as a Node.js service or an admin UI, can author modular DAG pipelines, validate them and trigger asynchronous runs over HTTP. This walkthrough starts the server with Ollama and OpenSearch and follows one full request lifecycle, from creating a project to reading per-node run statistics.
Preparing the backing services
The server orchestrates work, but embeddings and storage live elsewhere: here, Ollama for models and OpenSearch as the vector store. Start them first.
The quickest route is the project's QUICKSTART.md, which scripts Ollama, a vector database and the required containers in a few minutes. An AI coding agent such as Cursor, Claude Code or Copilot can also do the setup and verification; either way, confirm each service responds.
Choosing a vector store
This is an architectural decision. OpenSearch fits when you want hybrid search, mixing keyword matching with dense vectors. Milvus suits dedicated, highly scalable infrastructure for very large vector collections. This guide uses OpenSearch.
Starting the API server
Activate the virtual environment where the package is installed and run the bundled command:
source .venv/bin/activate
docling-pipelines-api
For hot reloading during development, run the app through uvicorn directly, bound to all interfaces on port 8080:
uvicorn docpipe.api.main:app --reload --host 0.0.0.0 --port 8080
Four URLs are then available:
http://localhost:8080/healthfor health checks;http://localhost:8080/api/v1/docsfor the interactive Swagger UI;http://localhost:8080/api/v1/redocfor ReDoc documentation;http://localhost:8080/api/v1/openapi.jsonfor the raw OpenAPI schema.
From TypeScript, you can feed the OpenAPI document to a client generator instead of hand-writing request types.
Environment variables that shape the server
Set these before launch to adjust behavior:
DS_LOG_LEVELsets log verbosity. It defaults toINFOand acceptsDEBUG,INFO,WARNINGorERROR.DOCPIPE_STORAGE_BACKENDchooses where metadata lives:duckdb(the default, local) orpostgres.CORS_ORIGINSlists origins allowed to call the API from a browser front end. It defaults tohttp://localhost:3000.OLLAMA_HOSTpoints embedding and LLM operators at Ollama. The default ishttp://localhost:11434; change it if Ollama runs on another port or machine.
Exploring endpoints in Swagger UI
Open http://localhost:8080/api/v1/docs in a browser; locally no login is needed. Expand an endpoint, press Try it out, and send the payloads below without writing a client.
The request lifecycle at a glance
A full round trip has seven calls, from project creation to the run summary:
1. Create Project ──▶ 2. Discover Operators ──▶ 3. Validate Flow ──▶ 4. Create Flow ──▶ 5. Execute Run ──▶ 6. Fetch Logs ──▶ 7. Summary
Step 1: create a project
Projects group flows and track their aggregated run statistics. Create one with POST /api/v1/projects:
{
"name": "Sample Docling Pipeline Project",
"description": "Flows for processing invoice documents",
"tags": ["finance", "pdf", "q3"]
}
The fields:
nameis a required string of 1 to 255 characters that uniquely identifies the project.descriptionis an optional short explanation of the project's purpose.tagsis an optional array of strings for categorization.
Success returns 201 Created with a generated project_id, a flow_count of zero, timestamps and an href to the resource:
{
"project_id": "84c718b2-5f36-4d2c-9a4f-561234bcde01",
"name": "Sample Docling Pipeline Project",
"description": "Flows for processing invoice documents",
"tags": ["finance", "pdf", "q3"],
"flow_count": 0,
"created_on": "2026-09-07T10:00:00Z",
"modified_on": "2026-09-07T10:00:00Z",
"href": "/api/v1/projects/84c718b2-5f36-4d2c-9a4f-561234bcde01"
}
Step 2: discover operators and models
Instead of guessing operator names and options, ask the server.
Operator metadata
GET /api/v1/operators/metadata returns a detailed schema for every operator, covering the 20+ built-in ones plus any external operators you have registered. The internal_features query parameter defaults to false.
In each entry, features describes the columns an operator produces, required_features the columns it needs from upstream, and attributes its options with types and defaults. The extract operator emits content and doc_id_hash; the chunker requires content:
{
"extract_operator": {
"label": "Extract Operator",
"category": "Extract",
"description": "Extracts structured content from documents using Docling",
"features": {
"content": {
"name": "Document Content",
"description": "The markdown content extracted from the document",
"type": "string",
"required": true
},
"doc_id_hash": {
"name": "Hash ID",
"description": "Hash ID of the document row",
"type": "string",
"required": true
}
},
"required_features": [],
"attributes": {
"text_extraction_mode": {
"type": "string",
"description": "Text extraction strategy (docling_library or docling_serve)",
"required": true,
"default": "docling_library"
},
"entity_extraction_mode": {
"type": "string",
"description": "Entity extraction strategy (litellm, watsonx, docling, or none)",
"required": false,
"default": "none"
}
}
},
"chunker": {
"label": "Chunker Operator",
"category": "Functional",
"description": "Splits documents into smaller semantic or fixed-size passages",
"features": {
"content": { "name": "Chunk Content", "type": "string", "required": true }
},
"required_features": ["content"],
"attributes": {
"chunker_type": {
"type": "string",
"description": "Chunking strategy (fixed, semantic, or hybrid)",
"required": false,
"default": "hybrid"
}
}
}
}
Comparing required_features with upstream outputs confirms, before you submit, that each node will get the columns it needs.
Provider models
GET /api/v1/providers/{provider}/models queries a live provider (ollama or watsonx), so a missing model or mistyped tag shows up before run time:
curl -s http://localhost:8080/api/v1/providers/ollama/models | jq
Each model is listed with name, tag, modification time and size in bytes:
{
"provider": "ollama",
"models": [
{
"name": "nomic-embed-text:latest",
"model": "nomic-embed-text:latest",
"modified_at": "2026-08-15T12:00:00Z",
"size": 274302450
},
{
"name": "llama3:8b",
"model": "llama3:8b",
"modified_at": "2026-08-20T14:30:00Z",
"size": 4661224676
}
]
}
Step 3: validate the flow before saving it
POST /api/v1/validation/validate_flow checks a DAG for bad parameters and broken dependencies without storing or running it.
global_config sets run-wide options (document column, local execution, in-memory storage, forced re-ingestion). flow lists nodes, each with a type, unique name, config and a depends_on array wiring it upstream. The pipeline:
- ingests PDF, TXT and DOCX files from
./sample_documentson the local filesystem; - extracts their text with the Docling library and skips entity extraction;
- splits content into 512-unit chunks with a 50-unit overlap;
- embeds each chunk through LiteLLM, pointing at Ollama's OpenAI-compatible endpoint with the
nomic-embed-textmodel; - writes the vectors to an OpenSearch index using the FAISS engine, HNSW and L2 distance with 768 dimensions, mapping pipeline features to index fields through
feature_mappings.
{
"flow_name": "complete-document-pipeline",
"global_config": {
"doc_column": "content",
"disable_validation": true,
"force_ingest": true,
"storage": "in-memory",
"execute_type": "local"
},
"flow": [
{
"type": "ingest_source",
"name": "ingest_source_filesystem",
"config": {
"provider": "filesystem",
"connection_params": {
"paths": ["./sample_documents"]
},
"include_filter": "pdf,txt,docx"
}
},
{
"type": "extract_operator",
"name": "extract_with_docling",
"config": {
"text_extraction": {
"provider": "docling_library",
"doc_column": "content"
},
"entity_extraction": {
"provider": "none"
}
},
"depends_on": ["ingest_source_filesystem"]
},
{
"type": "chunker",
"name": "simple_chunker",
"config": {
"chunk_type": "simple",
"chunk_size": 512,
"chunk_overlap": 50,
"retain_original_content": false
},
"depends_on": ["extract_with_docling"]
},
{
"type": "embeddings",
"name": "ollama_embeddings",
"config": {
"provider": "litellm",
"provider_config": {
"model_id": "openai/nomic-embed-text",
"api_base": "http://localhost:11434/v1",
"api_key": "<ollama-api-key>"
},
"embeddings_column": "embeddings",
"overlap_ratio": 0.1
},
"depends_on": ["simple_chunker"]
},
{
"type": "vectordb",
"name": "opensearch_vector_store",
"config": {
"provider": "opensearch",
"doc_id_column": "doc_id_hash",
"embeddings_column": "embeddings",
"vector_dimension": 768,
"create_index": true,
"provider_config": {
"index_name": "sample-documents-index",
"host": "localhost",
"port": 9200,
"username": "${OPENSEARCH_USERNAME}",
"password": "${OPENSEARCH_PASSWORD}",
"use_ssl": false,
"verify_certs": false,
"batch_size": 100,
"engine": "faiss",
"algorithm": "hnsw",
"space_type": "l2"
},
"feature_mappings": [
{ "feature_name": "doc_id_hash", "mapped_column_name": "pk" },
{ "feature_name": "content", "mapped_column_name": "text" },
{ "feature_name": "name", "mapped_column_name": "doc_name" },
{ "feature_name": "path", "mapped_column_name": "file_path" },
{ "feature_name": "embeddings", "mapped_column_name": "vector_embeddings" },
{ "feature_name": "chunk_id", "mapped_column_name": "chunk_id" },
{ "feature_name": "chunk_index", "mapped_column_name": "chunk_index" }
]
},
"depends_on": ["ollama_embeddings"]
}
]
}
Some details matter. vector_dimension (768) must match the embedding model's output size. Credentials appear as ${OPENSEARCH_USERNAME}-style and <ollama-api-key> placeholders; keep real secrets in environment variables or a secret store, not in stored flows. Disabling use_ssl and verify_certs is fine locally but not on a shared cluster. Also, the chunker uses chunk_type: "simple", whereas the metadata excerpt documents chunker_type with fixed, semantic and hybrid; check what your server returns.
A sound definition returns 200 OK, is_valid: true, empty errors and warnings, and the node count:
{
"is_valid": true,
"errors": [],
"warnings": [],
"node_count": 5
}
Step 4: register the flow
Store the validated definition with POST /api/v1/flows (is_elyra defaults to false). The body is the same DAG plus a description and tags:
{
"flow_name": "complete-document-pipeline",
"description": "Complete document processing pipeline: Ingest -> Extract -> Chunk -> Embed -> Store in OpenSearch",
"tags": ["docling", "embeddings", "opensearch"],
"global_config": {
"doc_column": "content",
"disable_validation": true,
"force_ingest": true,
"storage": "in-memory",
"execute_type": "local"
},
"flow": [
{
"type": "ingest_source",
"name": "ingest_source_filesystem",
"config": {
"provider": "filesystem",
"connection_params": { "paths": ["./sample_documents"] },
"include_filter": "pdf,txt,docx"
}
},
{
"type": "extract_operator",
"name": "extract_with_docling",
"config": {
"text_extraction": { "provider": "docling_library", "doc_column": "content" },
"entity_extraction": { "provider": "none" }
},
"depends_on": ["ingest_source_filesystem"]
},
{
"type": "chunker",
"name": "simple_chunker",
"config": {
"chunk_type": "simple",
"chunk_size": 512,
"chunk_overlap": 50,
"retain_original_content": false
},
"depends_on": ["extract_with_docling"]
},
{
"type": "embeddings",
"name": "ollama_embeddings",
"config": {
"provider": "litellm",
"provider_config": {
"model_id": "openai/nomic-embed-text",
"api_base": "http://localhost:11434/v1",
"api_key": "<ollama-api-key>"
},
"embeddings_column": "embeddings",
"overlap_ratio": 0.1
},
"depends_on": ["simple_chunker"]
},
{
"type": "vectordb",
"name": "opensearch_vector_store",
"config": {
"provider": "opensearch",
"doc_id_column": "doc_id_hash",
"embeddings_column": "embeddings",
"vector_dimension": 768,
"create_index": true,
"provider_config": {
"index_name": "sample-documents-index",
"host": "localhost",
"port": 9200,
"username": "${OPENSEARCH_USERNAME}",
"password": "${OPENSEARCH_PASSWORD}",
"use_ssl": false,
"verify_certs": false,
"batch_size": 100,
"engine": "faiss",
"algorithm": "hnsw",
"space_type": "l2"
},
"feature_mappings": [
{ "feature_name": "doc_id_hash", "mapped_column_name": "pk" },
{ "feature_name": "content", "mapped_column_name": "text" },
{ "feature_name": "name", "mapped_column_name": "doc_name" },
{ "feature_name": "path", "mapped_column_name": "file_path" },
{ "feature_name": "embeddings", "mapped_column_name": "vector_embeddings" },
{ "feature_name": "chunk_id", "mapped_column_name": "chunk_id" },
{ "feature_name": "chunk_index", "mapped_column_name": "chunk_index" }
]
},
"depends_on": ["ollama_embeddings"]
}
]
}
Two global settings matter here:
execute_type: "local"runs the pipeline inside the server process.force_ingest: truereprocesses previously ingested documents: handy while iterating, wasteful in steady state.
The reply is 201 Created with a new flow_id and the stored settings:
{
"flow_id": "e2c31e42-7719-482a-a53d-2490b830d971",
"flow_name": "complete-document-pipeline",
"description": "Complete document processing pipeline: Ingest -> Extract -> Chunk -> Embed -> Store in OpenSearch",
"tags": ["docling", "embeddings", "opensearch"],
"global_config": {
"doc_column": "content",
"disable_validation": true,
"force_ingest": true,
"storage": "in-memory",
"execute_type": "local"
},
"created_on": "2026-09-07T10:05:00Z",
"modified_on": "2026-09-07T10:05:00Z"
}
Step 5: start a run
Runs are asynchronous. POST /api/v1/job_runs takes the flow's ID and name:
{
"flow_id": "e2c31e42-7719-482a-a53d-2490b830d971",
"flow_name": "complete-document-pipeline"
}
It returns 201 Created at once with a job_run_id and status Starting, while work continues in the background:
{
"job_run_id": "99ecd646-bb8e-494a-a0c5-86b40c1d4f90",
"status": "Starting",
"message": "Job run created successfully"
}
Step 6: poll for status, metrics and logs
Poll GET /api/v1/job_runs/{job_run_id}, adding include_logs=true for per-node logs, until the status is terminal (such as Completed), ideally with growing delays.
The long 200 OK response contains node_sequence (execution order), job_stats (run-wide counters, with per-node node_stats and per-batch batch_node_stats) and a top-level node_metadata array. Here, two text files passed all five nodes, yielding four chunks, all indexed:
{
"node_sequence": [
"ca564648-1715-45c3-be67-7c654344a99c",
"49f285b0-3afa-40eb-8111-3ec6a57c4904",
"447e0511-5125-4ffb-8396-edb972976080",
"2636f8b6-ef9a-4805-bda0-9fcaa853e16a",
"0c773955-f1c7-4b68-b157-38966382cb9c"
],
"job_stats": {
"job_id": "9be63026-c08c-49c1-ba12-600838f3b22f",
"job_run_id": "13bd4219-7947-4977-bc69-1ae6628567d3",
"status": "Completed",
"message": "",
"start_time": 1788943163,
"end_time": 1788943166,
"duration": 3,
"heartbeat_timestamp": 0,
"total_docs": 2,
"processed_docs": 2,
"completed_docs": 2,
"failed_docs": 0,
"skipped_docs": 0,
"deleted_doc_count": 0,
"total_pages_processed": 2,
"page_type_stats": {
"txt": 2
},
"execution_time": 0,
"orchestrator": "Python",
"container_kind": null,
"container_id": "",
"flow_id": "unknown",
"user_id": null,
"account_id": null,
"user_entitlements": {},
"report_status": "NOT_AVAILABLE",
"report_generation_started_at": null,
"report_generation_completed_at": null,
"node_stats": {
"ca564648-1715-45c3-be67-7c654344a99c": {
"id": "ca564648-1715-45c3-be67-7c654344a99c",
"name": "extract_with_docling",
"node_status": "Completed",
"error": "",
"start_time": 1788943163,
"end_time": 1788943163,
"time_taken": 0,
"col_names": [
"content",
"doc_id_hash",
"document_format",
"id",
"metadata",
"modified_time",
"name",
"pages_processed",
"path",
"source_id"
],
"total_docs": 2,
"failed_docs": 0,
"skipped_docs": 0,
"docs_completed": 2,
"docs_completed_count": 2,
"node_metadata": {
"node_metadata": {
"failed_docs": [],
"processed_docs": 2,
"node_status": "Completed",
"total_pages_converted": 2,
"page_type_stats": {
"txt": 2
},
"failed_docs_count": 0,
"skipped_docs": [],
"documents_in_scope": 2,
"skipped_docs_count": 0,
"completed_docs_count": 2,
"Progress": "1 of 1 batches (100.0%) | Completed: 1"
},
"id": "ca564648-1715-45c3-be67-7c654344a99c",
"operator": "extract_with_docling"
},
"batch_id": null,
"batch_num": null
},
"49f285b0-3afa-40eb-8111-3ec6a57c4904": {
"id": "49f285b0-3afa-40eb-8111-3ec6a57c4904",
"name": "ingest_source_filesystem",
"node_status": "Completed",
"error": "",
"start_time": 1788943163,
"end_time": 1788943163,
"time_taken": 0,
"col_names": [
"id",
"name",
"document_format",
"metadata",
"source_id",
"path",
"modified_time"
],
"total_docs": 2,
"failed_docs": 0,
"skipped_docs": 0,
"docs_completed": 2,
"docs_completed_count": 2,
"node_metadata": {
"id": "49f285b0-3afa-40eb-8111-3ec6a57c4904",
"operator": "ingest_source_filesystem",
"node_metadata": {
"documents_in_scope": 2,
"processed_docs": 2,
"failed_docs_count": 0,
"failed_docs": [],
"skipped_docs_count": 0,
"skipped_docs": [],
"node_status": "Completed"
}
},
"batch_id": null,
"batch_num": null
},
"447e0511-5125-4ffb-8396-edb972976080": {
"id": "447e0511-5125-4ffb-8396-edb972976080",
"name": "simple_chunker",
"node_status": "Completed",
"error": "",
"start_time": 1788943163,
"end_time": 1788943164,
"time_taken": 1,
"col_names": [
"chunked_content",
"doc_id_hash",
"document_format",
"id",
"metadata",
"modified_time",
"name",
"pages_processed",
"path",
"source_id"
],
"total_docs": 2,
"failed_docs": 0,
"skipped_docs": 0,
"docs_completed": 2,
"docs_completed_count": 2,
"node_metadata": {
"node_metadata": {
"failed_docs": [],
"total_chunks": 4,
"processed_docs": 2,
"node_status": "Completed",
"failed_docs_count": 0,
"skipped_docs": [],
"documents_in_scope": 2,
"skipped_docs_count": 0,
"completed_docs_count": 2,
"Progress": "1 of 1 batches (100.0%) | Completed: 1"
},
"id": "447e0511-5125-4ffb-8396-edb972976080",
"operator": "simple_chunker"
},
"batch_id": null,
"batch_num": null
},
"2636f8b6-ef9a-4805-bda0-9fcaa853e16a": {
"id": "2636f8b6-ef9a-4805-bda0-9fcaa853e16a",
"name": "ollama_embeddings",
"node_status": "Completed",
"error": "",
"start_time": 1788943164,
"end_time": 1788943164,
"time_taken": 0,
"col_names": [
"chunked_content",
"doc_id_hash",
"document_format",
"embeddings",
"id",
"metadata",
"modified_time",
"name",
"pages_processed",
"path",
"source_id"
],
"total_docs": 2,
"failed_docs": 0,
"skipped_docs": 0,
"docs_completed": 2,
"docs_completed_count": 2,
"node_metadata": {
"node_metadata": {
"failed_docs": [],
"processed_docs": 2,
"node_status": "Completed",
"failed_docs_count": 0,
"skipped_docs": [],
"documents_in_scope": 2,
"skipped_docs_count": 0,
"completed_docs_count": 2,
"Progress": "1 of 1 batches (100.0%) | Completed: 1"
},
"id": "2636f8b6-ef9a-4805-bda0-9fcaa853e16a",
"operator": "ollama_embeddings"
},
"batch_id": null,
"batch_num": null
},
"0c773955-f1c7-4b68-b157-38966382cb9c": {
"id": "0c773955-f1c7-4b68-b157-38966382cb9c",
"name": "opensearch_vector_store",
"node_status": "Completed",
"error": "",
"start_time": 1788943164,
"end_time": 1788943165,
"time_taken": 1,
"col_names": [
"chunked_content",
"doc_id_hash",
"document_format",
"embeddings",
"id",
"metadata",
"modified_time",
"name",
"pages_processed",
"path",
"source_id"
],
"total_docs": 2,
"failed_docs": 0,
"skipped_docs": 0,
"docs_completed": 2,
"docs_completed_count": 2,
"node_metadata": {
"node_metadata": {
"failed_docs": [],
"chunks_indexed_successfully": 4,
"processed_docs": 2,
"total_chunks_to_index": 4,
"node_status": "Completed",
"chunks_failed_to_index": 0,
"number_of_batches": 1,
"failed_docs_count": 0,
"skipped_docs": [],
"documents_in_scope": 2,
"skipped_docs_count": 0,
"completed_docs_count": 2,
"Progress": "1 of 1 batches (100.0%) | Completed: 1"
},
"id": "0c773955-f1c7-4b68-b157-38966382cb9c",
"operator": "opensearch_vector_store"
},
"batch_id": null,
"batch_num": null
}
},
"batch_node_stats": {
"2636f8b6-ef9a-4805-bda0-9fcaa853e16a": {
"d9467158-dc5a-422e-92f7-d389776f8e50": {
"id": "2636f8b6-ef9a-4805-bda0-9fcaa853e16a",
"name": "ollama_embeddings",
"node_status": "Completed",
"error": "",
"start_time": 1788943164,
"end_time": 1788943164,
"time_taken": 0,
"col_names": [
"id",
"name",
"document_format",
"metadata",
"source_id",
"path",
"modified_time",
"pages_processed",
"chunked_content",
"embeddings",
"doc_id_hash"
],
"total_docs": 2,
"failed_docs": 0,
"skipped_docs": 0,
"docs_completed": 2,
"docs_completed_count": 2,
"node_metadata": {
"id": "2636f8b6-ef9a-4805-bda0-9fcaa853e16a",
"operator": "ollama_embeddings",
"node_metadata": {
"documents_in_scope": 2,
"processed_docs": 2,
"failed_docs_count": 0,
"failed_docs": [],
"skipped_docs_count": 0,
"skipped_docs": [],
"node_status": "Completed"
}
},
"batch_id": "d9467158-dc5a-422e-92f7-d389776f8e50",
"batch_num": 0
}
},
"0c773955-f1c7-4b68-b157-38966382cb9c": {
"d9467158-dc5a-422e-92f7-d389776f8e50": {
"id": "0c773955-f1c7-4b68-b157-38966382cb9c",
"name": "opensearch_vector_store",
"node_status": "Completed",
"error": "",
"start_time": 1788943164,
"end_time": 1788943165,
"time_taken": 1,
"col_names": [
"id",
"name",
"document_format",
"metadata",
"source_id",
"path",
"modified_time",
"pages_processed",
"chunked_content",
"embeddings",
"doc_id_hash"
],
"total_docs": 2,
"failed_docs": 0,
"skipped_docs": 0,
"docs_completed": 2,
"docs_completed_count": 2,
"node_metadata": {
"id": "0c773955-f1c7-4b68-b157-38966382cb9c",
"operator": "opensearch_vector_store",
"node_metadata": {
"documents_in_scope": 2,
"processed_docs": 2,
"failed_docs_count": 0,
"failed_docs": [],
"skipped_docs_count": 0,
"skipped_docs": [],
"node_status": "Completed",
"number_of_batches": 1,
"total_chunks_to_index": 4,
"chunks_indexed_successfully": 4,
"chunks_failed_to_index": 0
}
},
"batch_id": "d9467158-dc5a-422e-92f7-d389776f8e50",
"batch_num": 0
}
},
"447e0511-5125-4ffb-8396-edb972976080": {
"d9467158-dc5a-422e-92f7-d389776f8e50": {
"id": "447e0511-5125-4ffb-8396-edb972976080",
"name": "simple_chunker",
"node_status": "Completed",
"error": "",
"start_time": 1788943163,
"end_time": 1788943164,
"time_taken": 1,
"col_names": [
"id",
"name",
"document_format",
"metadata",
"source_id",
"path",
"modified_time",
"pages_processed",
"doc_id_hash",
"chunked_content"
],
"total_docs": 2,
"failed_docs": 0,
"skipped_docs": 0,
"docs_completed": 2,
"docs_completed_count": 2,
"node_metadata": {
"id": "447e0511-5125-4ffb-8396-edb972976080",
"operator": "simple_chunker",
"node_metadata": {
"documents_in_scope": 2,
"processed_docs": 2,
"failed_docs_count": 0,
"failed_docs": [],
"skipped_docs_count": 0,
"skipped_docs": [],
"node_status": "Completed",
"total_chunks": 4
}
},
"batch_id": "d9467158-dc5a-422e-92f7-d389776f8e50",
"batch_num": 0
}
},
"ca564648-1715-45c3-be67-7c654344a99c": {
"d9467158-dc5a-422e-92f7-d389776f8e50": {
"id": "ca564648-1715-45c3-be67-7c654344a99c",
"name": "extract_with_docling",
"node_status": "Completed",
"error": "",
"start_time": 1788943163,
"end_time": 1788943163,
"time_taken": 0,
"col_names": [
"id",
"name",
"document_format",
"metadata",
"source_id",
"path",
"modified_time",
"content",
"pages_processed",
"doc_id_hash"
],
"total_docs": 2,
"failed_docs": 0,
"skipped_docs": 0,
"docs_completed": 2,
"docs_completed_count": 2,
"node_metadata": {
"id": "ca564648-1715-45c3-be67-7c654344a99c",
"operator": "extract_with_docling",
"node_metadata": {
"documents_in_scope": 2,
"processed_docs": 2,
"failed_docs_count": 0,
"failed_docs": [],
"skipped_docs_count": 0,
"skipped_docs": [],
"node_status": "Completed",
"page_type_stats": {
"txt": 2
},
"total_pages_converted": 2
}
},
"batch_id": "d9467158-dc5a-422e-92f7-d389776f8e50",
"batch_num": 0
}
}
}
},
"node_metadata": [
{
"id": "ca564648-1715-45c3-be67-7c654344a99c",
"operator": "extract_with_docling",
"node_metadata": {
"failed_docs": [],
"processed_docs": 2,
"node_status": "Completed",
"total_pages_converted": 2,
"page_type_stats": {
"txt": 2
},
"failed_docs_count": 0,
"skipped_docs": [],
"documents_in_scope": 2,
"skipped_docs_count": 0,
"completed_docs_count": 2,
"Progress": "1 of 1 batches (100.0%) | Completed: 1"
}
},
{
"id": "49f285b0-3afa-40eb-8111-3ec6a57c4904",
"operator": "ingest_source_filesystem",
"node_metadata": {
"documents_in_scope": 2,
"processed_docs": 2,
"failed_docs_count": 0,
"failed_docs": [],
"skipped_docs_count": 0,
"skipped_docs": [],
"node_status": "Completed"
}
},
{
"id": "447e0511-5125-4ffb-8396-edb972976080",
"operator": "simple_chunker",
"node_metadata": {
"failed_docs": [],
"total_chunks": 4,
"processed_docs": 2,
"node_status": "Completed",
"failed_docs_count": 0,
"skipped_docs": [],
"documents_in_scope": 2,
"skipped_docs_count": 0,
"completed_docs_count": 2,
"Progress": "1 of 1 batches (100.0%) | Completed: 1"
}
},
{
"id": "2636f8b6-ef9a-4805-bda0-9fcaa853e16a",
"operator": "ollama_embeddings",
"node_metadata": {
"failed_docs": [],
"processed_docs": 2,
"node_status": "Completed",
"failed_docs_count": 0,
"skipped_docs": [],
"documents_in_scope": 2,
"skipped_docs_count": 0,
"completed_docs_count": 2,
"Progress": "1 of 1 batches (100.0%) | Completed: 1"
}
},
{
"id": "0c773955-f1c7-4b68-b157-38966382cb9c",
"operator": "opensearch_vector_store",
"node_metadata": {
"failed_docs": [],
"chunks_indexed_successfully": 4,
"processed_docs": 2,
"total_chunks_to_index": 4,
"node_status": "Completed",
"chunks_failed_to_index": 0,
"number_of_batches": 1,
"failed_docs_count": 0,
"skipped_docs": [],
"documents_in_scope": 2,
"skipped_docs_count": 0,
"completed_docs_count": 2,
"Progress": "1 of 1 batches (100.0%) | Completed: 1"
}
}
]
}
This sample is not fully consistent: the job_run_id in job_stats differs from step 5's ID and flow_id reads unknown. Correlate runs by the ID you received at start.
What to read first in the response
job_statsgives high-level counters such astotal_docs,processed_docs,completed_docsandduration.node_statsshows each step's duration and status; check it first when a run is slow or fails.- With
include_logs=true, formatted logs are added at the response root, keyed by each node's name (from the ingest node through to the OpenSearch store).
Interpreting node metadata
Each operator reports telemetry under node_metadata: throughput, errors and conversions per stage. The main attributes:
documents_in_scope: an integer count of raw documents the ingest operator found and loaded; your baseline.processed_docs: an integer count this operator handled successfully in the run; equal todocuments_in_scopewhen nothing fails.failed_docs_count: an integer count of documents that errored here.failed_docs: an array whose entries hold anid, the original filename, an errorreasonand adocument_url(blank when files come from the local disk).skipped_docs_count: an integer count of documents intentionally bypassed, say by deduplication, branch logic or incremental runs.skipped_docs: an array shaped likefailed_docs, wherereasonexplains the skip, such as"Filtered by branch condition".page_type_stats: reported only by the extract operator, a dictionary from file extension to parsed page count, for example{"pdf": 45, "docx": 12, "txt": 3}.- Also extract-only,
total_pages_convertedis the integer sum of pages across formats, using native counts for digital files and OCR counts for scans.
A quick health check: compare documents_in_scope at ingest with processed_docs at the last node; failed_docs and skipped_docs explain any gap.
Step 7: review flows and run history for a project
GET /api/v1/projects/{project_id}/flows lists a project's flows with a run summary (total runs, last run ID, status and start time, per-status counts), paginated by total_count, offset and limit:
{
"flows": [
{
"flow_id": "e2c31e42-7719-482a-a53d-2490b830d971",
"name": "complete-document-pipeline",
"tags": ["docling", "embeddings", "opensearch"],
"created_on": "2026-09-07T10:05:00Z",
"job_run_summary": {
"total_runs": 1,
"last_run_id": "99ecd646-bb8e-494a-a0c5-86b40c1d4f90",
"last_run_status": "Completed",
"last_run_start_time": 1788775200000,
"status_counts": { "Completed": 1 }
}
}
],
"total_count": 1,
"offset": 0,
"limit": 100
}
In this sample last_run_start_time looks like milliseconds while job run start_time values are seconds, so normalize before comparing.
Key takeaways
The API offers a headless interface for end-to-end document processing. A few habits make it reliable:
- Use
/operators/metadataand/providers/{provider}/modelsto discover operator fields and available model tags programmatically instead of hard-coding them. - Run every DAG definition through
/validation/validate_flowto catch wrong parameter types and broken node connections before execution. - Treat runs as asynchronous jobs: keep the
job_run_id, poll withinclude_logs=true, and readnode_statsandnode_metadatato locate failures by stage. - Keep credentials out of stored flow definitions and verify that
vector_dimensionmatches your embedding model.
To try it, start Ollama and OpenSearch (or Milvus), run docling-pipelines-api, open the Swagger UI, and work through the sequence: project, flow, validation, run, logs.