Quality-Aware RAG Ingestion: Routing Documents by Readability

Moving a customer-support knowledge base into AI search usually goes wrong at the same point: every file gets the same treatment. A realistic corpus is a mixed bag. It holds tidy FAQs, structured product guides, dense developer manuals, duplicated legal text, and a few corrupted files with no readable words at all.

Pour all of that into one conversational RAG system and retrieval quality drops. A customer who wants a simple how-to gets chunks of an engineering manual. Sending garbage text or system logs to an expensive model for summaries or translation is also money thrown away.

The fix is to route documents by quality. A two-tier design looks at each file, removes empty pages, measures readability and then splits the corpus:

1. Tier 1, consumer search. Highly readable documents get structural metadata, such as word counts, to support customer-facing conversational search.
2. Tier 2, developer and reference search. Dense technical material goes to its own index for advanced technical questions.

docling-pipelines makes this practical. It's an open-source, MIT-licensed framework backed by IBM that builds and runs document-processing graphs straight from JSON configuration. No custom Python glue is needed, because operators hand documents to each other as in-memory Apache Arrow tables, so intermediate transforms and routing checks cost almost nothing in CPU and memory.

The shape of the pipeline

Everything lives in one JSON flow file that describes a directed acyclic graph:

1. ingest reads PDF, TXT, DOCX and HTML files from disk.
2. extract pulls structured text out with the Docling library.
3. detect_language identifies each document's language.
4. dedup uses content hashes to remove duplicate rows.
5. doc_quality computes structural metrics, such as total word count.
6. readability calculates text-complexity scores.
7. quality_branching routes on the Flesch Reading Ease score: above 60 goes down the high-quality branch, 60 or lower goes down the low-quality branch.
8. The high-quality path runs ml_enrichment_high_quality, embeddings_high_quality and vectordb_high_quality. It adds ML-based linguistic features, creates vector embeddings and writes into a high_quality_docs index.
9. The low-quality path runs sql_filter_low_quality, embeddings_low_quality and vectordb_low_quality. It removes empty rows, embeds what's left and writes into low_quality_docs.

Profiling before the split

Each document needs a profile before it can be routed. After ingest reads the payloads and extract turns the layout into text, every later operator works as a transformer: it inspects the table and adds new columns to the same in-memory Arrow table. Cheap CPU-only checks come first: language, duplicates and text complexity.

Language detection

Later ML models and readability formulas need to know the language, so detection comes first.

{
      "name": "detect_language",
      "type": "lang_detect",
      "config": {
        "doc_column": "content"
      },
      "depends_on": [
        "extract"
      ]
    }

The operator reads the content column and appends lang_name (an ISO 639-1 code) and lang_score. Placing it right after extraction means every language-sensitive step downstream can use its output.

Deduplication

Duplicate documents waste embedding compute and skew search results.

{
      "name": "dedup",
      "type": "ededup",
      "config": {
        "doc_column": "content",
        "doc_id_column": "doc_id_hash"
      },
      "depends_on": [
        "detect_language"
      ]
    }

It refers to the content hash stored in doc_id_hash and to the raw text in content, then removes repeated rows from the table. Doing this early spares the later steps from useless work.

Document quality

Basic structural checks catch blank pages and odd layouts.

{
      "name": "doc_quality",
      "type": "doc_quality",
      "config": {
        "doc_column": "content",
        "doc_content_column": "content",
        "text_lang": "en"
      },
      "depends_on": [
        "dedup"
      ]
    }

This adds 11 structural columns, all prefixed docq_. The one that matters here is docq_total_words, the total word count for each document.

Readability

Readability formulas judge difficulty from word length, syllables and sentence structure.

{
      "name": "readability",
      "type": "readability",
      "config": {
        "doc_column": "content",
        "readability_score_list": [
          "flesch_reading_ease",
          "flesch_kincaid_grade",
          "gunning_fog",
          "difficult_words",
          "dale_chall_readability_score"
        ]
      },
      "depends_on": [
        "doc_quality"
      ]
    }

The readability_score_list array tells the operator which metrics to compute and append as columns. Routing relies on the Flesch Reading Ease score, which runs from 0 to 100:

• Above 60 means plain, accessible language that suits a general help-center audience.
• 60 or lower means dense, technical prose that assumes specialist knowledge.

The branching step

The branching operator cuts the Arrow dataset into separate processing paths.

{
      "name": "quality_branching",
      "type": "branching",
      "config": {
        "branches": {
          "high_quality_branch": {
            "link_name": "High Quality Documents",
            "criteria_json": {
              "criteria_list": [
                {
                  "variable": "flesch_reading_ease",
                  "operator": ">",
                  "value": 60
                }
              ],
              "logical_operator": "AND"
            }
          },
          "low_quality_branch": {
            "link_name": "Low Quality Documents",
            "criteria_json": {
              "criteria_list": [
                {
                  "variable": "flesch_reading_ease",
                  "operator": "<=",
                  "value": 60
                }
              ],
              "logical_operator": "AND"
            }
          }
        }
      },
      "depends_on": [
        "readability"
      ]
    }

Routing is controlled by a declarative criteria_json block with four fields:

• variable names the column to test, for example flesch_reading_ease.
• operator gives the comparison, such as > or <=.
• value is the threshold, here 60.
• logical_operator joins several conditions with AND or OR.

With flesch_reading_ease > 60 as the rule, the engine evaluates the conditions natively, splits the in-memory table and runs each downstream branch.

High-quality path: enrich, embed, index

Documents sent down high_quality_branch receive linguistic ML enrichment first.

{
      "name": "ml_enrichment_high_quality",
      "type": "ml_enrichment",
      "config": {
        "doc_column": "content",
        "lang_column": "lang_name",
        "output_column_prefix": "ml_",
        "error_column_name": "enrichment_error"
      },
      "depends_on": [
        "quality_branching.high_quality_branch"
      ]
    }

ml_enrichment parses the text with language-aware models and produces metadata such as sentence counts, syllable properties and noun lists, all aimed at better retrieval. It depends on the lang_name column from the detection step. Because this is the CPU-heavy part of the flow, it runs only on readable text. The output_column_prefix setting prefixes the new columns with ml_, which includes ml_num_words.

Next, the text becomes 2560-dimensional embeddings. The model is openai/granite4:latest, served by a local Ollama server through LiteLLM:

{
      "name": "embeddings_high_quality",
      "type": "embeddings",
      "config": {
        "embeddings_column": "embeddings",
        "overlap_ratio": 0.2,
        "doc_column": "content",
        "provider_config": {
          "api_base": "http://localhost:11434/v1",
          "api_key": "${OLLAMA_API_KEY}",
          "model_id": "openai/granite4:latest"
        },
        "provider": "litellm"
      },
      "depends_on": [
        "ml_enrichment_high_quality"
      ]
    }

Last, documents are indexed into OpenSearch, with feature_mappings tying Arrow columns to OpenSearch fields:

{
      "name": "vectordb_high_quality",
      "type": "vectordb",
      "config": {
        "provider": "opensearch",
        "doc_id_column": "doc_id_hash",
        "embeddings_column": "embeddings",
        "vector_dimension": 2560,
        "create_index": "true",
        "provider_config": {
          "index_name": "high_quality_docs",
          "host": "localhost",
          "port": 9200,
          "username": "${OPENSEARCH_USERNAME}",
          "password": "${OPENSEARCH_PASSWORD}",
          "use_ssl": false,
          "verify_certs": false,
          "batch_size": 100,
          "engine": "faiss",
          "algorithm": "hnsw",
          "space_type": "l2"
        },
        "feature_mappings": [
          {"feature_name": "doc_id_hash", "mapped_column_name": "pk"},
          {"feature_name": "name", "mapped_column_name": "doc_name"},
          {"feature_name": "content", "mapped_column_name": "text"},
          {"feature_name": "embeddings", "mapped_column_name": "vector_embeddings"},
          {"feature_name": "lang_name", "mapped_column_name": "language"},
          {"feature_name": "flesch_reading_ease", "mapped_column_name": "readability_score"},
          {"feature_name": "ml_num_words", "mapped_column_name": "word_count"}
        ]
      },
      "depends_on": [
        "embeddings_high_quality"
      ]
    }

Low-quality path: filter, embed, index

Documents on low_quality_branch skip the costly ml_enrichment step. Some of them may contain no readable words after extraction, for example corrupted files or blank scans, and embedding an empty string only adds noise to the vector space. A filter removes them:

{
      "name": "sql_filter_low_quality",
      "type": "sql_filter",
      "config": {
        "criteria_json": {
          "criteria_list": [
            {
              "variable": "docq_total_words",
              "operator": ">",
              "value": 0
            }
          ],
          "logical_operator": "AND"
        }
      },
      "depends_on": [
        "quality_branching.low_quality_branch"
      ]
    }

The filter checks the docq_total_words column from doc_quality and drops any document with zero words.

The survivors, mostly technical material, are embedded with the same granite4 model and written to a separate OpenSearch index:

{
      "name": "vectordb_low_quality",
      "type": "vectordb",
      "config": {
        "provider": "opensearch",
        "doc_id_column": "doc_id_hash",
        "embeddings_column": "embeddings",
        "vector_dimension": 2560,
        "create_index": "true",
        "provider_config": {
          "index_name": "low_quality_docs",
          "host": "localhost",
          "port": 9200,
          "username": "${OPENSEARCH_USERNAME}",
          "password": "${OPENSEARCH_PASSWORD}",
          "use_ssl": false,
          "verify_certs": false,
          "batch_size": 100,
          "engine": "faiss",
          "algorithm": "hnsw",
          "space_type": "l2"
        },
        "feature_mappings": [
          {"feature_name": "doc_id_hash", "mapped_column_name": "pk"},
          {"feature_name": "name", "mapped_column_name": "doc_name"},
          {"feature_name": "content", "mapped_column_name": "text"},
          {"feature_name": "embeddings", "mapped_column_name": "vector_embeddings"},
          {"feature_name": "lang_name", "mapped_column_name": "language"},
          {"feature_name": "flesch_reading_ease", "mapped_column_name": "readability_score"},
          {"feature_name": "difficult_words", "mapped_column_name": "difficult_words"}
        ]
      },
      "depends_on": [
        "embeddings_low_quality"
      ]
    }

Since this branch skipped enrichment, word_count doesn't exist for these documents. The mapping uses the computed difficult_words count instead, which helps with technical queries.

Two indexes, two query styles

Separate indexes per readability tier let each audience search differently.

Help-center search

For end users, the query hits high_quality_docs. It keeps only easy-to-read guides (readability_score > 60) with real substance (word_count > 100):

curl -XGET "http://localhost:9200/high_quality_docs/_search" \
  -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must": [
        {
          "knn": {
            "vector_embeddings": {
              "vector": [0.015, -0.023, 0.004],
              "k": 10
            }
          }
        }
      ],
      "filter": [
        {
          "range": {
            "readability_score": {
              "gt": 60
            }
          }
        },
        {
          "range": {
            "word_count": {
              "gt": 100
            }
          }
        }
      ]
    }
  }
}'

Developer reference search

For expert users hunting specifications, the query targets low_quality_docs. It looks at the complex material and keeps segments with many hard technical terms (difficult_words > 5):

curl -XGET "http://localhost:9200/low_quality_docs/_search" \
  -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must": [
        {
          "knn": {
            "vector_embeddings": {
              "vector": [0.015, -0.023, 0.004],
              "k": 10
            }
          }
        }
      ],
      "filter": [
        {
          "range": {
            "difficult_words": {
              "gt": 5
            }
          }
        }
      ]
    }
  }
}'

Running it

The repository ships a setup script that creates a virtual environment, installs dependencies, starts local OpenSearch containers and registers the Ollama models.

Step 1: set up the environment

Clone the repo and run the script:

git clone https://github.com/IBM/docling-pipelines.git
cd docling-pipelines
./scripts/setup_docling_pipelines_environment.sh

Or install straight from PyPI:

pip install docling-pipelines

Step 2: set environment variables

export OLLAMA_API_KEY=ollama
export OPENSEARCH_USERNAME=admin
export OPENSEARCH_PASSWORD=<your-password>

Step 3: run the flow

Launch the pipeline from the command line:

docling-pipelines --flow-file sample_flows/advanced/branching_quality_routing.json

What the run prints

The output traces the flow, with each transformation and routing decision logged:

$ docling-pipelines --flow-file sample_flows/advanced/branching_quality_routing.json
[INFO] [ingest] Loading files from tests/fixtures/customer_support_docs/
[INFO] [ingest] Ingested 12 files.
[INFO] [extract] Running extract_operator using docling_library...
[INFO] [extract] Extracted text from 12 documents.
[INFO] [detect_language] Running lang_detect...
[INFO] [detect_language] Complete. lang_name assigned.
[INFO] [dedup] Running ededup...
[INFO] [dedup] Removed 2 duplicate documents. 10 documents remaining.
[INFO] [doc_quality] Running doc_quality...
[INFO] [doc_quality] Computed docq_* features for 10 documents.
[INFO] [readability] Running readability...
[INFO] [readability] Computed readability features for 10 documents.
[INFO] [quality_branching] Branching 10 documents...
[INFO] [quality_branching] Routed 6 documents to link 'High Quality Documents' (flesch_reading_ease > 60).
[INFO] [quality_branching] Routed 4 documents to link 'Low Quality Documents' (flesch_reading_ease <= 60).
[INFO] [ml_enrichment_high_quality] Running ml_enrichment on 6 documents...
[INFO] [embeddings_high_quality] Generating embeddings with model 'openai/granite4:latest'...
[INFO] [vectordb_high_quality] Uploaded 6 documents to OpenSearch index 'high_quality_docs'.
[INFO] [sql_filter_low_quality] Running sql_filter on 4 documents...
[INFO] [sql_filter_low_quality] Filtered out 0 empty documents. 4 documents remaining.
[INFO] [embeddings_low_quality] Generating embeddings with model 'openai/granite4:latest'...
[INFO] [vectordb_low_quality] Uploaded 4 documents to OpenSearch index 'low_quality_docs'.
[INFO] Flow execution finished successfully in 4.23s.

Moving quality metrics and routing rules out of custom Python and into a declarative config makes ingestion fast and easy to maintain. Processing cost follows document value, and the RAG application receives clean, tailored indexes for each audience.

The branching idea works with any metric computed earlier in the flow. Change the criteria to route by language, PII risk score or a custom classifier label. Because pipeline logic stays separate from orchestration code, teams can grow the pipeline without growing the maintenance burden.