首页 / 文章

RAGDocument ProcessingREST APIOpenSearchOllama

通过HTTP驱动Docling管道：从项目设置到索引分块

逐步了解 Docling Pipelines REST API：启动服务器、发现操作符、验证并运行数据摄取 DAG，以及查看其执行监控信息。

发布者：REACTAPP.TOP阅读约 15 分钟3355 词

来源: **A Hands-On Guide to the Docling Pipelines REST API**

将PDF和Word文件所在的文件夹转换为可搜索的向量块通常需要借助专门的代码来完成提取、分块、嵌入和索引等操作。Docling Pipelines将这些步骤封装为基于FastAPI的REST服务器后的操作函数，这样任何客户端——比如Node.js服务或管理界面——都可以构建模块化的有向无环图管道，对其进行验证，并通过HTTP触发异步执行。本指南将使用Ollama和OpenSearch启动服务器，完整演示从创建项目到查看每个节点的执行统计数据的整个请求生命周期。

准备后端服务

服务器负责协调各项任务，但嵌入处理和数据存储则发生在其他地方：这里使用Ollama来管理模型，OpenSearch作为向量存储库。请先启动这两个服务。

最快的方法是通过项目的QUICKSTART.md文档，该文档可在几分钟内自动配置好 Ollama、向量数据库以及所需的容器。像 Cursor、Claude Code 或 Copilot 这样的 AI 编程助手也能完成设置与验证工作；无论哪种方式，都需要确认各项服务都能正常响应。

选择向量存储库

这属于架构层面的决策。如果需要结合关键词匹配与密集向量进行混合搜索，OpenSearch 是合适的选择；而对于规模极大的向量集合，则更适合使用具备高度可扩展性的专用基础设施 Milvus。本指南采用 OpenSearch。

启动 API 服务器

激活已安装该软件包的虚拟环境，然后运行附带的命令：

source .venv/bin/activate
docling-pipelines-api

如需在开发过程中进行热重载，可直接通过 uvicorn 运行应用，将其绑定到端口 8080 的所有接口上：

uvicorn docpipe.api.main:app --reload --host 0.0.0.0 --port 8080

此时会有四个可用 URL：

  • http://localhost:8080/health 用于健康检查；
  • http://localhost:8080/api/v1/docs 用于交互式 Swagger UI；
  • http://localhost:8080/api/v1/redoc 用于 ReDoc 文档；
  • http://localhost:8080/api/v1/openapi.json 用于获取原始的 OpenAPI 架构文件。

使用 TypeScript 时，可以直接将 OpenAPI 文档传递给客户端生成器，而无需手动编写请求类型。

影响服务器行为的环境变量

在启动前设置这些变量即可调整应用行为：

  • DS_LOG_LEVEL用于设置日志详细程度。默认值为INFO，可接受的值为DEBUG、INFO、WARNING或ERROR。
  • DOCPIPE_STORAGE_BACKEND决定元数据存储的位置：默认为本地的duckdb，或postgres。
  • CORS_ORIGINS列出了允许从浏览器前端调用API的源地址，默认值为http://localhost:3000。
  • OLLAMA_HOST用于指定嵌入模型和大型语言模型运算符的连接地址，默认为http://localhost:11434；如果Ollama运行在其他端口或机器上，则需进行相应修改。

在Swagger UI中查看端点

在浏览器中打开 http://localhost:8080/api/v1/docs；本地使用无需登录。展开某个接口端点，点击 Try it out，即可无需编写客户端代码直接发送下面的请求数据。

请求生命周期概览

从项目创建到运行汇总，一个完整的往返过程包含七次调用：

1. Create Project ──▶ 2. Discover Operators ──▶ 3. Validate Flow ──▶ 4. Create Flow ──▶ 5. Execute Run ──▶ 6. Fetch Logs ──▶ 7. Summary

步骤1：创建项目

项目用于对流程进行分组并统计其整体运行数据。可通过 POST /api/v1/projects 创建项目：

{
  "name": "Sample Docling Pipeline Project",
  "description": "Flows for processing invoice documents",
  "tags": ["finance", "pdf", "q3"]
}

字段说明：

  • name 是必填的字符串，长度为1到255个字符，用于唯一标识该项目。
  • description 是可选的简短说明，用于描述项目的用途。
  • tags 是可选的字符串数组，用于对项目进行分类。

操作成功时会返回201 Created状态码，同时包含生成的project_id、值为零的flow_count、时间戳以及指向该资源的href：

{
  "project_id": "84c718b2-5f36-4d2c-9a4f-561234bcde01",
  "name": "Sample Docling Pipeline Project",
  "description": "Flows for processing invoice documents",
  "tags": ["finance", "pdf", "q3"],
  "flow_count": 0,
  "created_on": "2026-09-07T10:00:00Z",
  "modified_on": "2026-09-07T10:00:00Z",
  "href": "/api/v1/projects/84c718b2-5f36-4d2c-9a4f-561234bcde01"
}

第2步：发现操作符和模型

无需猜测操作符的名称和选项，可直接向服务器查询。

操作符元数据

GET /api/v1/operators/metadata会返回每个操作符的详细结构信息，涵盖20多种内置操作符以及您注册的任何外部操作符。internal_features查询参数的默认值为false。

在每个条目中，features描述了操作符生成的列，required_features描述了其从上游需要的列，而attributes则列出了带有类型和默认值的选项。提取操作符会输出content和doc_id_hash；分块操作符则需要content：

{
  "extract_operator": {
    "label": "Extract Operator",
    "category": "Extract",
    "description": "Extracts structured content from documents using Docling",
    "features": {
      "content": {
        "name": "Document Content",
        "description": "The markdown content extracted from the document",
        "type": "string",
        "required": true
      },
      "doc_id_hash": {
        "name": "Hash ID",
        "description": "Hash ID of the document row",
        "type": "string",
        "required": true
      }
    },
    "required_features": [],
    "attributes": {
      "text_extraction_mode": {
        "type": "string",
        "description": "Text extraction strategy (docling_library or docling_serve)",
        "required": true,
        "default": "docling_library"
      },
      "entity_extraction_mode": {
        "type": "string",
        "description": "Entity extraction strategy (litellm, watsonx, docling, or none)",
        "required": false,
        "default": "none"
      }
    }
  },
  "chunker": {
    "label": "Chunker Operator",
    "category": "Functional",
    "description": "Splits documents into smaller semantic or fixed-size passages",
    "features": {
      "content": { "name": "Chunk Content", "type": "string", "required": true }
    },
    "required_features": ["content"],
    "attributes": {
      "chunker_type": {
        "type": "string",
        "description": "Chunking strategy (fixed, semantic, or hybrid)",
        "required": false,
        "default": "hybrid"
      }
    }
  }
}

在提交之前，通过将required_features与上游输出进行比较，可以确认每个节点都能获得所需的列。

提供者模型

GET /api/v1/providers/{provider}/models会查询正在运行的提供者（如ollama或watsonx），因此如果模型缺失或标签输入有误，会在运行前就显现出来：

curl -s http://localhost:8080/api/v1/providers/ollama/models | jq

每个模型都会列出其名称、标签、修改时间以及以字节为单位的大小：

{
  "provider": "ollama",
  "models": [
    {
      "name": "nomic-embed-text:latest",
      "model": "nomic-embed-text:latest",
      "modified_at": "2026-08-15T12:00:00Z",
      "size": 274302450
    },
    {
      "name": "llama3:8b",
      "model": "llama3:8b",
      "modified_at": "2026-08-20T14:30:00Z",
      "size": 4661224676
    }
  ]
}

步骤3：在保存之前验证流程

POST /api/v1/validation/validate_flow 可在不存储或运行 DAG 的情况下检查其中是否存在错误参数及断开的依赖关系。

global_config 用于设置全局运行选项（文档列、本地执行、内存存储、强制重新处理）。flow 列出各个节点，每个节点包含 type、唯一的 name、config 以及指向上游节点的 depends_on 数组。处理流程如下：

  • 从本地文件系统中的 ./sample_documents 获取 PDF、TXT 和 DOCX 格式的文件；
    • 使用 Docling 库提取这些文件的文本，跳过实体抽取步骤； 将内容分割成大小为 512 单位、重叠部分为 50 单位的块； 通过 LiteLLM 嵌入每个数据块，使用 nomic-embed-text 模型并指向 Ollama 的 OpenAI 兼容端点。
  • 使用 FAISS 引擎将向量写入 OpenSearch 索引，采用 HNSW 算法及 768 维度的 L2 距离，通过 feature_mappings 将特征管道中的字段映射到索引字段。
    • {
  "flow_name": "complete-document-pipeline",
  "global_config": {
    "doc_column": "content",
    "disable_validation": true,
    "force_ingest": true,
    "storage": "in-memory",
    "execute_type": "local"
  },
  "flow": [
    {
      "type": "ingest_source",
      "name": "ingest_source_filesystem",
      "config": {
        "provider": "filesystem",
        "connection_params": {
          "paths": ["./sample_documents"]
        },
        "include_filter": "pdf,txt,docx"
      }
    },
    {
      "type": "extract_operator",
      "name": "extract_with_docling",
      "config": {
        "text_extraction": {
          "provider": "docling_library",
          "doc_column": "content"
        },
        "entity_extraction": {
          "provider": "none"
        }
      },
      "depends_on": ["ingest_source_filesystem"]
    },
    {
      "type": "chunker",
      "name": "simple_chunker",
      "config": {
        "chunk_type": "simple",
        "chunk_size": 512,
        "chunk_overlap": 50,
        "retain_original_content": false
      },
      "depends_on": ["extract_with_docling"]
    },
    {
      "type": "embeddings",
      "name": "ollama_embeddings",
      "config": {
        "provider": "litellm",
        "provider_config": {
          "model_id": "openai/nomic-embed-text",
          "api_base": "http://localhost:11434/v1",
          "api_key": "<ollama-api-key>"
        },
        "embeddings_column": "embeddings",
        "overlap_ratio": 0.1
      },
      "depends_on": ["simple_chunker"]
    },
    {
      "type": "vectordb",
      "name": "opensearch_vector_store",
      "config": {
        "provider": "opensearch",
        "doc_id_column": "doc_id_hash",
        "embeddings_column": "embeddings",
        "vector_dimension": 768,
        "create_index": true,
        "provider_config": {
          "index_name": "sample-documents-index",
          "host": "localhost",
          "port": 9200,
          "username": "${OPENSEARCH_USERNAME}",
          "password": "${OPENSEARCH_PASSWORD}",
          "use_ssl": false,
          "verify_certs": false,
          "batch_size": 100,
          "engine": "faiss",
          "algorithm": "hnsw",
          "space_type": "l2"
        },
        "feature_mappings": [
          { "feature_name": "doc_id_hash",  "mapped_column_name": "pk" },
          { "feature_name": "content",      "mapped_column_name": "text" },
          { "feature_name": "name",         "mapped_column_name": "doc_name" },
          { "feature_name": "path",         "mapped_column_name": "file_path" },
          { "feature_name": "embeddings",   "mapped_column_name": "vector_embeddings" },
          { "feature_name": "chunk_id",     "mapped_column_name": "chunk_id" },
          { "feature_name": "chunk_index",  "mapped_column_name": "chunk_index" }
        ]
      },
      "depends_on": ["ollama_embeddings"]
    }
  ]
}

    有些细节很重要。vector_dimension（768）必须与嵌入模型的输出尺寸一致。凭证以 ${OPENSEARCH_USERNAME} 格式或 <ollama-api-key> 占位符的形式出现；真正的密钥应存储在环境变量或密钥管理系统中，而非保存在流程文件中。在本地环境下关闭 use_ssl 和 verify_certs 没问题，但在共享集群上则不可行。此外，分块器使用的是 chunk_type: "simple"，而元数据说明中提到的 chunker_type 则包括 fixed、semantic 和 hybrid 这几种类型；请查看服务器返回的实际值。

    有效的定义会返回 200 OK 状态、is_valid: true、空的 errors 和 warnings 字段，以及节点数量：

    {
  "is_valid": true,
  "errors": [],
  "warnings": [],
  "node_count": 5
}

    第4步：注册流程

    使用 POST /api/v1/flows 保存经过验证的定义（is_elyra 的默认值为 false）。请求体包含相同的有向无环图结构，以及描述信息和标签：

    {
  "flow_name": "complete-document-pipeline",
  "description": "Complete document processing pipeline: Ingest -> Extract -> Chunk -> Embed -> Store in OpenSearch",
  "tags": ["docling", "embeddings", "opensearch"],

  "global_config": {
    "doc_column": "content",
    "disable_validation": true,
    "force_ingest": true,
    "storage": "in-memory",
    "execute_type": "local"
  },
  "flow": [
    {
      "type": "ingest_source",
      "name": "ingest_source_filesystem",
      "config": {
        "provider": "filesystem",
        "connection_params": { "paths": ["./sample_documents"] },
        "include_filter": "pdf,txt,docx"
      }
    },
    {
      "type": "extract_operator",
      "name": "extract_with_docling",
      "config": {
        "text_extraction": { "provider": "docling_library", "doc_column": "content" },
        "entity_extraction": { "provider": "none" }
      },
      "depends_on": ["ingest_source_filesystem"]
    },
    {
      "type": "chunker",
      "name": "simple_chunker",
      "config": {
        "chunk_type": "simple",
        "chunk_size": 512,
        "chunk_overlap": 50,
        "retain_original_content": false
      },
      "depends_on": ["extract_with_docling"]
    },
    {
      "type": "embeddings",
      "name": "ollama_embeddings",
      "config": {
        "provider": "litellm",
        "provider_config": {
          "model_id": "openai/nomic-embed-text",
          "api_base": "http://localhost:11434/v1",
          "api_key": "<ollama-api-key>"
        },
        "embeddings_column": "embeddings",
        "overlap_ratio": 0.1
      },
      "depends_on": ["simple_chunker"]
    },
    {
      "type": "vectordb",
      "name": "opensearch_vector_store",
      "config": {
        "provider": "opensearch",
        "doc_id_column": "doc_id_hash",
        "embeddings_column": "embeddings",
        "vector_dimension": 768,
        "create_index": true,
        "provider_config": {
          "index_name": "sample-documents-index",
          "host": "localhost",
          "port": 9200,
          "username": "${OPENSEARCH_USERNAME}",
          "password": "${OPENSEARCH_PASSWORD}",
          "use_ssl": false,
          "verify_certs": false,
          "batch_size": 100,
          "engine": "faiss",
          "algorithm": "hnsw",
          "space_type": "l2"
        },
        "feature_mappings": [
          { "feature_name": "doc_id_hash", "mapped_column_name": "pk" },
          { "feature_name": "content", "mapped_column_name": "text" },
          { "feature_name": "name", "mapped_column_name": "doc_name" },
          { "feature_name": "path", "mapped_column_name": "file_path" },
          { "feature_name": "embeddings", "mapped_column_name": "vector_embeddings" },
          { "feature_name": "chunk_id", "mapped_column_name": "chunk_id" },
          { "feature_name": "chunk_index", "mapped_column_name": "chunk_index" }
        ]
      },
      "depends_on": ["ollama_embeddings"]
    }
  ]
}

    此处有两个重要的全局设置：

    • execute_type: "local" 会在服务器进程内部运行管道。
    • force_ingest: true 会重新处理之前已导入的文档：在迭代时很有用，但在稳定运行状态下则会造成资源浪费。

    响应为 201 Created，其中包含新的 flow_id 以及已保存的设置：

    {
  "flow_id": "e2c31e42-7719-482a-a53d-2490b830d971",
  "flow_name": "complete-document-pipeline",
  "description": "Complete document processing pipeline: Ingest -> Extract -> Chunk -> Embed -> Store in OpenSearch",
  "tags": ["docling", "embeddings", "opensearch"],
  "global_config": {
    "doc_column": "content",
    "disable_validation": true,
    "force_ingest": true,
    "storage": "in-memory",
    "execute_type": "local"
  },
  "created_on": "2026-09-07T10:05:00Z",
  "modified_on": "2026-09-07T10:05:00Z"
}

    第5步：启动运行

    运行是异步的。POST /api/v1/job_runs需要传入流程的ID和名称：

    {
  "flow_id": "e2c31e42-7719-482a-a53d-2490b830d971",
  "flow_name": "complete-document-pipeline"
}

    它会立即返回201 Created状态，并附带job_run_id以及Starting的运行状态，同时任务会在后台继续执行：

    {
  "job_run_id": "99ecd646-bb8e-494a-a0c5-86b40c1d4f90",
  "status": "Starting",
  "message": "Job run created successfully"
}

    第6步：轮询状态、指标和日志

    不断调用GET /api/v1/job_runs/{job_run_id}，并添加include_logs=true参数以获取每个节点的日志，直到状态变为最终状态（如Completed），建议逐渐增加轮询间隔。

    漫长的 200 OK 响应中包含 node_sequence（执行顺序）、job_stats（全局计数器，以及每个节点的 node_stats 和每批次的 batch_node_stats）以及一个顶层 node_metadata 数组。此处，两个文本文件通过了所有五个节点的处理，生成了四个数据块，且均有索引：

    {
  "node_sequence": [
    "ca564648-1715-45c3-be67-7c654344a99c",
    "49f285b0-3afa-40eb-8111-3ec6a57c4904",
    "447e0511-5125-4ffb-8396-edb972976080",
    "2636f8b6-ef9a-4805-bda0-9fcaa853e16a",
    "0c773955-f1c7-4b68-b157-38966382cb9c"
  ],
  "job_stats": {
    "job_id": "9be63026-c08c-49c1-ba12-600838f3b22f",
    "job_run_id": "13bd4219-7947-4977-bc69-1ae6628567d3",
    "status": "Completed",
    "message": "",
    "start_time": 1788943163,
    "end_time": 1788943166,
    "duration": 3,
    "heartbeat_timestamp": 0,
    "total_docs": 2,
    "processed_docs": 2,
    "completed_docs": 2,
    "failed_docs": 0,
    "skipped_docs": 0,
    "deleted_doc_count": 0,
    "total_pages_processed": 2,
    "page_type_stats": {
      "txt": 2
    },
    "execution_time": 0,
    "orchestrator": "Python",
    "container_kind": null,
    "container_id": "",
    "flow_id": "unknown",
    "user_id": null,
    "account_id": null,
    "user_entitlements": {},
    "report_status": "NOT_AVAILABLE",
    "report_generation_started_at": null,
    "report_generation_completed_at": null,
    "node_stats": {
      "ca564648-1715-45c3-be67-7c654344a99c": {
        "id": "ca564648-1715-45c3-be67-7c654344a99c",
        "name": "extract_with_docling",
        "node_status": "Completed",
        "error": "",
        "start_time": 1788943163,
        "end_time": 1788943163,
        "time_taken": 0,
        "col_names": [
          "content",
          "doc_id_hash",
          "document_format",
          "id",
          "metadata",
          "modified_time",
          "name",
          "pages_processed",
          "path",
          "source_id"
        ],
        "total_docs": 2,
        "failed_docs": 0,
        "skipped_docs": 0,
        "docs_completed": 2,
        "docs_completed_count": 2,
        "node_metadata": {
          "node_metadata": {
            "failed_docs": [],
            "processed_docs": 2,
            "node_status": "Completed",
            "total_pages_converted": 2,
            "page_type_stats": {
              "txt": 2
            },
            "failed_docs_count": 0,
            "skipped_docs": [],
            "documents_in_scope": 2,
            "skipped_docs_count": 0,
            "completed_docs_count": 2,
            "Progress": "1 of 1 batches (100.0%) | Completed: 1"
          },
          "id": "ca564648-1715-45c3-be67-7c654344a99c",
          "operator": "extract_with_docling"
        },
        "batch_id": null,
        "batch_num": null
      },
      "49f285b0-3afa-40eb-8111-3ec6a57c4904": {
        "id": "49f285b0-3afa-40eb-8111-3ec6a57c4904",
        "name": "ingest_source_filesystem",
        "node_status": "Completed",
        "error": "",
        "start_time": 1788943163,
        "end_time": 1788943163,
        "time_taken": 0,
        "col_names": [
          "id",
          "name",
          "document_format",
          "metadata",
          "source_id",
          "path",
          "modified_time"
        ],
        "total_docs": 2,
        "failed_docs": 0,
        "skipped_docs": 0,
        "docs_completed": 2,
        "docs_completed_count": 2,
        "node_metadata": {
          "id": "49f285b0-3afa-40eb-8111-3ec6a57c4904",
          "operator": "ingest_source_filesystem",
          "node_metadata": {
            "documents_in_scope": 2,
            "processed_docs": 2,
            "failed_docs_count": 0,
            "failed_docs": [],
            "skipped_docs_count": 0,
            "skipped_docs": [],
            "node_status": "Completed"
          }
        },
        "batch_id": null,
        "batch_num": null
      },
      "447e0511-5125-4ffb-8396-edb972976080": {
        "id": "447e0511-5125-4ffb-8396-edb972976080",
        "name": "simple_chunker",
        "node_status": "Completed",
        "error": "",
        "start_time": 1788943163,
        "end_time": 1788943164,
        "time_taken": 1,
        "col_names": [
          "chunked_content",
          "doc_id_hash",
          "document_format",
          "id",
          "metadata",
          "modified_time",
          "name",
          "pages_processed",
          "path",
          "source_id"
        ],
        "total_docs": 2,
        "failed_docs": 0,
        "skipped_docs": 0,
        "docs_completed": 2,
        "docs_completed_count": 2,
        "node_metadata": {
          "node_metadata": {
            "failed_docs": [],
            "total_chunks": 4,
            "processed_docs": 2,
            "node_status": "Completed",
            "failed_docs_count": 0,
            "skipped_docs": [],
            "documents_in_scope": 2,
            "skipped_docs_count": 0,
            "completed_docs_count": 2,
            "Progress": "1 of 1 batches (100.0%) | Completed: 1"
          },
          "id": "447e0511-5125-4ffb-8396-edb972976080",
          "operator": "simple_chunker"
        },
        "batch_id": null,
        "batch_num": null
      },
      "2636f8b6-ef9a-4805-bda0-9fcaa853e16a": {
        "id": "2636f8b6-ef9a-4805-bda0-9fcaa853e16a",
        "name": "ollama_embeddings",
        "node_status": "Completed",
        "error": "",
        "start_time": 1788943164,
        "end_time": 1788943164,
        "time_taken": 0,
        "col_names": [
          "chunked_content",
          "doc_id_hash",
          "document_format",
          "embeddings",
          "id",
          "metadata",
          "modified_time",
          "name",
          "pages_processed",
          "path",
          "source_id"
        ],
        "total_docs": 2,
        "failed_docs": 0,
        "skipped_docs": 0,
        "docs_completed": 2,
        "docs_completed_count": 2,
        "node_metadata": {
          "node_metadata": {
            "failed_docs": [],
            "processed_docs": 2,
            "node_status": "Completed",
            "failed_docs_count": 0,
            "skipped_docs": [],
            "documents_in_scope": 2,
            "skipped_docs_count": 0,
            "completed_docs_count": 2,
            "Progress": "1 of 1 batches (100.0%) | Completed: 1"
          },
          "id": "2636f8b6-ef9a-4805-bda0-9fcaa853e16a",
          "operator": "ollama_embeddings"
        },
        "batch_id": null,
        "batch_num": null
      },
      "0c773955-f1c7-4b68-b157-38966382cb9c": {
        "id": "0c773955-f1c7-4b68-b157-38966382cb9c",
        "name": "opensearch_vector_store",
        "node_status": "Completed",
        "error": "",
        "start_time": 1788943164,
        "end_time": 1788943165,
        "time_taken": 1,
        "col_names": [
          "chunked_content",
          "doc_id_hash",
          "document_format",
          "embeddings",
          "id",
          "metadata",
          "modified_time",
          "name",
          "pages_processed",
          "path",
          "source_id"
        ],
        "total_docs": 2,
        "failed_docs": 0,
        "skipped_docs": 0,
        "docs_completed": 2,
        "docs_completed_count": 2,
        "node_metadata": {
          "node_metadata": {
            "failed_docs": [],
            "chunks_indexed_successfully": 4,
            "processed_docs": 2,
            "total_chunks_to_index": 4,
            "node_status": "Completed",
            "chunks_failed_to_index": 0,
            "number_of_batches": 1,
            "failed_docs_count": 0,
            "skipped_docs": [],
            "documents_in_scope": 2,
            "skipped_docs_count": 0,
            "completed_docs_count": 2,
            "Progress": "1 of 1 batches (100.0%) | Completed: 1"
          },
          "id": "0c773955-f1c7-4b68-b157-38966382cb9c",
          "operator": "opensearch_vector_store"
        },
        "batch_id": null,
        "batch_num": null
      }
    },
    "batch_node_stats": {
      "2636f8b6-ef9a-4805-bda0-9fcaa853e16a": {
        "d9467158-dc5a-422e-92f7-d389776f8e50": {
          "id": "2636f8b6-ef9a-4805-bda0-9fcaa853e16a",
          "name": "ollama_embeddings",
          "node_status": "Completed",
          "error": "",
          "start_time": 1788943164,
          "end_time": 1788943164,
          "time_taken": 0,
          "col_names": [
            "id",
            "name",
            "document_format",
            "metadata",
            "source_id",
            "path",
            "modified_time",
            "pages_processed",
            "chunked_content",
            "embeddings",
            "doc_id_hash"
          ],
          "total_docs": 2,
          "failed_docs": 0,
          "skipped_docs": 0,
          "docs_completed": 2,
          "docs_completed_count": 2,
          "node_metadata": {
            "id": "2636f8b6-ef9a-4805-bda0-9fcaa853e16a",
            "operator": "ollama_embeddings",
            "node_metadata": {
              "documents_in_scope": 2,
              "processed_docs": 2,
              "failed_docs_count": 0,
              "failed_docs": [],
              "skipped_docs_count": 0,
              "skipped_docs": [],
              "node_status": "Completed"
            }
          },
          "batch_id": "d9467158-dc5a-422e-92f7-d389776f8e50",
          "batch_num": 0
        }
      },
      "0c773955-f1c7-4b68-b157-38966382cb9c": {
        "d9467158-dc5a-422e-92f7-d389776f8e50": {
          "id": "0c773955-f1c7-4b68-b157-38966382cb9c",
          "name": "opensearch_vector_store",
          "node_status": "Completed",
          "error": "",
          "start_time": 1788943164,
          "end_time": 1788943165,
          "time_taken": 1,
          "col_names": [
            "id",
            "name",
            "document_format",
            "metadata",
            "source_id",
            "path",
            "modified_time",
            "pages_processed",
            "chunked_content",
            "embeddings",
            "doc_id_hash"
          ],
          "total_docs": 2,
          "failed_docs": 0,
          "skipped_docs": 0,
          "docs_completed": 2,
          "docs_completed_count": 2,
          "node_metadata": {
            "id": "0c773955-f1c7-4b68-b157-38966382cb9c",
            "operator": "opensearch_vector_store",
            "node_metadata": {
              "documents_in_scope": 2,
              "processed_docs": 2,
              "failed_docs_count": 0,
              "failed_docs": [],
              "skipped_docs_count": 0,
              "skipped_docs": [],
              "node_status": "Completed",
              "number_of_batches": 1,
              "total_chunks_to_index": 4,
              "chunks_indexed_successfully": 4,
              "chunks_failed_to_index": 0
            }
          },
          "batch_id": "d9467158-dc5a-422e-92f7-d389776f8e50",
          "batch_num": 0
        }
      },
      "447e0511-5125-4ffb-8396-edb972976080": {
        "d9467158-dc5a-422e-92f7-d389776f8e50": {
          "id": "447e0511-5125-4ffb-8396-edb972976080",
          "name": "simple_chunker",
          "node_status": "Completed",
          "error": "",
          "start_time": 1788943163,
          "end_time": 1788943164,
          "time_taken": 1,
          "col_names": [
            "id",
            "name",
            "document_format",
            "metadata",
            "source_id",
            "path",
            "modified_time",
            "pages_processed",
            "doc_id_hash",
            "chunked_content"
          ],
          "total_docs": 2,
          "failed_docs": 0,
          "skipped_docs": 0,
          "docs_completed": 2,
          "docs_completed_count": 2,
          "node_metadata": {
            "id": "447e0511-5125-4ffb-8396-edb972976080",
            "operator": "simple_chunker",
            "node_metadata": {
              "documents_in_scope": 2,
              "processed_docs": 2,
              "failed_docs_count": 0,
              "failed_docs": [],
              "skipped_docs_count": 0,
              "skipped_docs": [],
              "node_status": "Completed",
              "total_chunks": 4
            }
          },
          "batch_id": "d9467158-dc5a-422e-92f7-d389776f8e50",
          "batch_num": 0
        }
      },
      "ca564648-1715-45c3-be67-7c654344a99c": {
        "d9467158-dc5a-422e-92f7-d389776f8e50": {
          "id": "ca564648-1715-45c3-be67-7c654344a99c",
          "name": "extract_with_docling",
          "node_status": "Completed",
          "error": "",
          "start_time": 1788943163,
          "end_time": 1788943163,
          "time_taken": 0,
          "col_names": [
            "id",
            "name",
            "document_format",
            "metadata",
            "source_id",
            "path",
            "modified_time",
            "content",
            "pages_processed",
            "doc_id_hash"
          ],
          "total_docs": 2,
          "failed_docs": 0,
          "skipped_docs": 0,
          "docs_completed": 2,
          "docs_completed_count": 2,
          "node_metadata": {
            "id": "ca564648-1715-45c3-be67-7c654344a99c",
            "operator": "extract_with_docling",
            "node_metadata": {
              "documents_in_scope": 2,
              "processed_docs": 2,
              "failed_docs_count": 0,
              "failed_docs": [],
              "skipped_docs_count": 0,
              "skipped_docs": [],
              "node_status": "Completed",
              "page_type_stats": {
                "txt": 2
              },
              "total_pages_converted": 2
            }
          },
          "batch_id": "d9467158-dc5a-422e-92f7-d389776f8e50",
          "batch_num": 0
        }
      }
    }
  },
  "node_metadata": [
    {
      "id": "ca564648-1715-45c3-be67-7c654344a99c",
      "operator": "extract_with_docling",
      "node_metadata": {
        "failed_docs": [],
        "processed_docs": 2,
        "node_status": "Completed",
        "total_pages_converted": 2,
        "page_type_stats": {
          "txt": 2
        },
        "failed_docs_count": 0,
        "skipped_docs": [],
        "documents_in_scope": 2,
        "skipped_docs_count": 0,
        "completed_docs_count": 2,
        "Progress": "1 of 1 batches (100.0%) | Completed: 1"
      }
    },
    {
      "id": "49f285b0-3afa-40eb-8111-3ec6a57c4904",
      "operator": "ingest_source_filesystem",
      "node_metadata": {
        "documents_in_scope": 2,
        "processed_docs": 2,
        "failed_docs_count": 0,
        "failed_docs": [],
        "skipped_docs_count": 0,
        "skipped_docs": [],
        "node_status": "Completed"
      }
    },
    {
      "id": "447e0511-5125-4ffb-8396-edb972976080",
      "operator": "simple_chunker",
      "node_metadata": {
        "failed_docs": [],
        "total_chunks": 4,
        "processed_docs": 2,
        "node_status": "Completed",
        "failed_docs_count": 0,
        "skipped_docs": [],
        "documents_in_scope": 2,
        "skipped_docs_count": 0,
        "completed_docs_count": 2,
        "Progress": "1 of 1 batches (100.0%) | Completed: 1"
      }
    },
    {
      "id": "2636f8b6-ef9a-4805-bda0-9fcaa853e16a",
      "operator": "ollama_embeddings",
      "node_metadata": {
        "failed_docs": [],
        "processed_docs": 2,
        "node_status": "Completed",
        "failed_docs_count": 0,
        "skipped_docs": [],
        "documents_in_scope": 2,
        "skipped_docs_count": 0,
        "completed_docs_count": 2,
        "Progress": "1 of 1 batches (100.0%) | Completed: 1"
      }
    },
    {
      "id": "0c773955-f1c7-4b68-b157-38966382cb9c",
      "operator": "opensearch_vector_store",
      "node_metadata": {
        "failed_docs": [],
        "chunks_indexed_successfully": 4,
        "processed_docs": 2,
        "total_chunks_to_index": 4,
        "node_status": "Completed",
        "chunks_failed_to_index": 0,
        "number_of_batches": 1,
        "failed_docs_count": 0,
        "skipped_docs": [],
        "documents_in_scope": 2,
        "skipped_docs_count": 0,
        "completed_docs_count": 2,
        "Progress": "1 of 1 batches (100.0%) | Completed: 1"
      }
    }
  ]
}

    该示例并不一致：job_stats 中的 job_run_id 与第5步的ID不同，且 flow_id 的值为 unknown。请根据启动时获得的ID来关联各次运行。

    响应中应首先查看的内容

    • job_stats 提供了诸如 total_docs、processed_docs、completed_docs 和 duration 等高级计数器信息。
  • node_stats会显示每一步的耗时与状态；当运行速度过慢或失败时，应首先查看该信息。
  • 若设置include_logs=true，格式化后的日志会添加到响应的根目录中，以每个节点的名称（从数据导入节点到OpenSearch存储节点）作为键。

    • 解读节点元数据

    每个处理节点都会在node_metadata中报告相关指标：各阶段的吞吐量、错误数量及转换情况。主要属性包括：

    • documents_in_scope：数据导入节点发现并加载的原始文档数量，可作为基准值。
    • processed_docs：该节点在本次运行中成功处理的文档数量；若没有出现错误，则该值与documents_in_scope相同。
    • failed_docs_count：在此步骤出现错误的文档数量。
  • failed_docs：一个数组，其元素包含id、原始文件name、错误reason以及document_url（若文件来自本地磁盘则该值为空）。
  • skipped_docs_count：因重复删除、分支逻辑或增量处理等原因而被刻意跳过的文档数量，为整数。
  • skipped_docs：结构与failed_docs相同的数组，其中的reason字段说明了跳过的原因，例如"Filtered by branch condition"。
  • page_type_stats：仅由提取操作符生成，是一个以文件扩展名为键、解析后的页面数量为值的字典，例如{"pdf": 45, "docx": 12, "txt": 3}。
  • 同样仅用于提取数据，total_pages_converted表示各格式页面数的整数总和，数字文件以原生计数为准，扫描件则以OCR识别结果为准。

    • 简易健康检查：将导入时的documents_in_scope与最终节点的processed_docs进行比较；failed_docs和skipped_docs可解释其中的差异。

    第7步：查看项目的流程及运行历史

    GET /api/v1/projects/{project_id}/flows可列出项目的所有流程及运行摘要（总运行次数、上次运行ID、状态与开始时间、各状态的计数），并通过total_count、offset和limit参数进行分页：

    {
  "flows": [
    {
      "flow_id": "e2c31e42-7719-482a-a53d-2490b830d971",
      "name": "complete-document-pipeline",
      "tags": ["docling", "embeddings", "opensearch"],
      "created_on": "2026-09-07T10:05:00Z",
      "job_run_summary": {
        "total_runs": 1,
        "last_run_id": "99ecd646-bb8e-494a-a0c5-86b40c1d4f90",
        "last_run_status": "Completed",
        "last_run_start_time": 1788775200000,
        "status_counts": { "Completed": 1 }
      }
    }
  ],
  "total_count": 1,
  "offset": 0,
  "limit": 100
}

    在此示例中，last_run_start_time的单位似乎是毫秒，而任务运行的start_time单位为秒，因此在比较之前需先进行标准化处理。

    核心要点

    该 API 提供无头接口以实现端到端的文档处理。以下几项习惯可确保其稳定性：

    • 使用 /operators/metadata 和 /providers/{provider}/models 以编程方式动态获取操作符字段及可用模型标签，而非直接硬编码。
    • 通过 /validation/validate_flow 对每个 DAG 定义进行验证，以便在执行前发现参数类型错误及节点连接问题。
    • 将运行过程视为异步任务：保留 job_run_id，使用 include_logs=true 进行轮询，并读取 node_stats 与 node_metadata 以按阶段定位故障点。
    • 不要将凭证存储在流程定义中，并确认 vector_dimension 与您的嵌入模型匹配。

    要尝试使用它，先启动 Ollama 和 OpenSearch（或 Milvus），运行 docling-pipelines-api，打开 Swagger UI，然后按照项目、流程、验证、运行、日志的顺序操作。

    ← 全部文章发送简报