Data Cleaning for Production RAG: Encoding, OCR, Dedup and PII

Picture a bank that has put a knowledge assistant on top of 200,000 internal and regulatory documents. Ingestion runs fine. PDFs parse, scanned AML circulars go through OCR, SharePoint files arrive on schedule, and tables are pulled out separately. Nothing seems to be missing from the corpus.

Yet answers are uneven. A compliance officer asks which documents are required for enhanced due diligence on a high-risk corporate customer. The retriever returns five chunks. Three hold the relevant KYC policy, and two carry nearly identical text from an older version of the document. Each chunk begins with the same banner:

CONFIDENTIAL - INTERNAL USE ONLY
Group Financial Crime Compliance
Page 47 of 132

One chunk also contains the sentence "AII high-risk corporate customers must undergo enhanced due diIigence." Look closely: All has turned into AII and diligence into diIigence, because OCR mixed up a lowercase l and a capital I.

Elsewhere in the same file a non-breaking space survived extraction, a Windows-1252 apostrophe was decoded wrongly, and that footer shows up 132 times because it sat on every page. The archived copy of the policy sits in the index next to the approved one, since SharePoint kept both.

Technically the ingestion pipeline worked. In operational terms the corpus is dirty, and engineers building their first RAG system often underestimate how much that difference matters.

Why dirty data hides

The damage is usually subtle enough that nothing crashes. A malformed byte sequence won't make a vector database throw. A duplicated paragraph doesn't make an embedding API fail. A repeated footer sets off no alert, and an OCR slip from 1 to l won't stop the indexing job. The pipeline embeds everything cheerfully, and the bill arrives later as retrieval failures.

Duplicated documents occupy several slots in the top-k results. Repeated boilerplate creates artificial similarity between unrelated chunks. OCR errors change critical entities, percentages, account numbers and regulatory limits. Encoding artefacts add tokens with no business meaning. By the time a bad answer appears, every downstream component may be doing its job: the retriever returned what was stored, the reranker ranked what it was given, and the LLM worked from whatever context reached it. The fault arose earlier than all three.

Take an AML policy that says transactions above EUR 10,000 call for additional source-of-funds checks. Once bad OCR and a faulty encoding conversion have run, what gets indexed reads "EUR l0,000". A person spots the error instantly. An embedding model still treats the sentence as close to concepts like transaction thresholds and source-of-funds checks, so retrieval works perfectly well, which is the unsettling part. The corrupted statement reaches the LLM because semantic retrieval forgives textual imperfection. The system finds the right policy section with the wrong regulatory value inside it.

That's why cleaning can't be reduced to text.strip().lower().

Where cleaning belongs in a RAG pipeline

Cleaning sits between document ingestion and chunking, and the sequence is deliberate. You don't want to measure document similarity before fixing obvious encoding corruption. You don't want to normalize text aggressively before detecting its language. You don't want to delete repeated legal clauses just because they resemble boilerplate. And low-confidence OCR corrections shouldn't quietly rewrite compliance obligations.

Once noisy text has been split into chunks and embedded, cleaning gets much harder. Duplicate paragraphs have already produced duplicate vectors, broken words have shaped chunk boundaries, repeated headers have used up token space, and OCR errors are baked into the semantic representation. Cleaning belongs ahead of chunking, not as a patch applied after quality starts to slip.

Encoding normalization: repair text first

Enterprise collections rarely share one encoding history. A modern policy may be UTF-8, a customer correspondence archive may be Windows-1252 and older exports Latin-1. Text copied between SharePoint, email, Word and legacy document systems may have been encoded and decoded several times before it reaches ingestion. The outcome looks like this:

Customer’s identity must be verified before account opening.
The customer must provide proof of address.
Enhanced Due Diligence â€" High Risk Customers

This corruption is called mojibake, and it happens when bytes written in one encoding are read as another. A person sees at once that customer’s is broken. For a retrieval pipeline, the corrupted token reaches the tokenizer differently from customer's, and at scale such inconsistencies fragment the vocabulary and add needless variation to the embedding space.

Be careful about how far repair goes. A reckless function decides that any character beyond ASCII is junk, yet a European payments corpus legitimately holds Société Générale, Deutsche Bundesbank, €10,000 and Überweisung. Strip the accents and symbols and real information disappears. The goal is not an ASCII-only text; it's consistent Unicode that keeps its meaning.

Repairing Unicode with ftfy

You don't have to catalogue every mojibake pattern yourself, because ftfy already repairs the usual Unicode and encoding mistakes. A production cleaning stage can begin there:

import re
import unicodedata
from dataclasses import dataclass, field
import ftfy

@dataclass
class TextCleaningResult:
    original_text: str
    cleaned_text: str
    transformations: list[str] = field(default_factory=list)
    quality_flags: list[str] = field(default_factory=list)
def normalize_encoding(text: str) -> TextCleaningResult:
    """
    Repair common encoding problems while preserving meaningful Unicode.
    Suitable for policy, AML/KYC, payment, and regulatory documents where
    accented names, currency symbols, and multilingual text must survive.
    """
    transformations: list[str] = []
    quality_flags: list[str] = []
    if not text:
        return TextCleaningResult(original_text=text, cleaned_text=text)
    cleaned = text
    # Repair mojibake and common Unicode encoding mistakes.
    repaired = ftfy.fix_text(cleaned)
    if repaired != cleaned:
        transformations.append("ftfy_encoding_repair")
        cleaned = repaired
    # NFC preserves characters while producing a consistent Unicode
    # representation. Safer than aggressively stripping accents.
    normalized = unicodedata.normalize("NFC", cleaned)
    if normalized != cleaned:
        transformations.append("unicode_nfc_normalization")
        cleaned = normalized
    # Replace non-breaking spaces with normal spaces.
    if "\u00a0" in cleaned:
        cleaned = cleaned.replace("\u00a0", " ")
        transformations.append("non_breaking_space_normalization")
    # Remove zero-width characters that frequently leak from PDFs,
    # web pages, and copied Office content.
    zero_width_chars = {"\u200b", "\u200c", "\u200d", "\ufeff"}
    if any(char in cleaned for char in zero_width_chars):
        cleaned = "".join(char for char in cleaned if char not in zero_width_chars)
        transformations.append("zero_width_character_removal")
    # Remove control characters; preserve newline and tab because
    # they may still carry document structure needed downstream.
    cleaned_without_controls = "".join(
        char for char in cleaned
        if char in "\n\t" or unicodedata.category(char) != "Cc"
    )
    if cleaned_without_controls != cleaned:
        transformations.append("control_character_removal")
        cleaned = cleaned_without_controls
    # Normalize horizontal whitespace without flattening paragraphs.
    whitespace_normalized = re.sub(r"[ \t]+", " ", cleaned)
    whitespace_normalized = re.sub(r"\n{3,}", "\n\n", whitespace_normalized)
    if whitespace_normalized != cleaned:
        transformations.append("whitespace_normalization")
        cleaned = whitespace_normalized
    cleaned = cleaned.strip()
    # Keep suspicious replacement characters observable rather than silently deleting them.
    if "\ufffd" in cleaned:
        quality_flags.append("unicode_replacement_character_detected")
    return TextCleaningResult(
        original_text=text,
        cleaned_text=cleaned,
        transformations=transformations,
        quality_flags=quality_flags,
    )

Note what the function leaves alone. It doesn't lowercase everything, strip accents from Société Générale, remove the euro sign or flatten the document into a single line. Those omissions are intentional.

Why heavy-handed cleaning hurts

Many older NLP pipelines were designed for bag-of-words models or TF-IDF, and they lowercased, stripped punctuation, removed stop words and stemmed. Applying that recipe blindly before modern embeddings throws away useful information. Consider what it would do to this sentence:

Customer must NOT be classified as low risk.

Financial text offers even more delicate examples. EUR 10,000 and EUR 100,000 look alike in letters yet differ greatly in operational meaning, and Article 6(1)(a) and Article 6(1)(b) have identical structure but different legal force. The working rule is to normalize representation aggressively and meaning conservatively. Turning customer’s into customer's is a change of form that restores what the writer meant. Dropping €, removing accents from a customer's name, or auto-converting l0% to 10% edits facts the business might rely on.

Boilerplate removal: structure, not keywords

Headers and footers are not the same kind of problem as encoding noise. Page 47 of 132 repeating is clearly boilerplate, but a compliance requirement that appears in several policy sections may be repeated on purpose and carry legal weight. How often a line occurs can't say which of the two you're looking at.

A better signal combines repetition with position. Headers cluster at the top of pages and footers at the bottom, and page numbers shift slightly while keeping the same structural pattern:

KYC Policy | Version 7.2 | Page 41 of 132
KYC Policy | Version 7.2 | Page 42 of 132

Exact-match logic sees two different strings, while structure-aware logic sees two instances of one template. That's one reason boilerplate detection works best while page boundaries still exist, so avoid flattening a parsed document into a giant string right after ingestion.

import re
from collections import Counter
from dataclasses import dataclass

@dataclass
class BoilerplatePattern:
    normalized_text: str
    occurrences: int
    page_ratio: float
    position: str
def normalize_boilerplate_candidate(line: str) -> str:
    """
    Normalize variable fields so structurally identical headers and
    footers can be compared across pages.
    """
    normalized = line.strip()
    normalized = re.sub(
        r"\bpage\s+\d+\s+of\s+\d+\b", "page <n> of <n>",
        normalized, flags=re.IGNORECASE,
    )
    normalized = re.sub(r"\bpage\s+\d+\b", "page <n>", normalized, flags=re.IGNORECASE)
    normalized = re.sub(r"\s+", " ", normalized)
    return normalized.casefold()
def detect_repeated_marginal_text(
    pages: list[dict],
    margin_lines: int = 3,
    min_page_ratio: float = 0.6,
) -> list[BoilerplatePattern]:
    """
    Detect repeated text in page-header or page-footer regions.
    A candidate is considered boilerplate only when it appears in the same
    marginal position across a substantial fraction of the document.
    """
    if not pages:
        return []
    header_counts: Counter = Counter()
    footer_counts: Counter = Counter()
    for page in pages:
        lines = [line.strip() for line in page["text"].splitlines() if line.strip()]
        if not lines:
            continue
        header_counts.update(normalize_boilerplate_candidate(l) for l in lines[:margin_lines])
        footer_counts.update(normalize_boilerplate_candidate(l) for l in lines[-margin_lines:])
    total_pages = len(pages)
    patterns: list[BoilerplatePattern] = []
    for position, counts in (("header", header_counts), ("footer", footer_counts)):
        for normalized_text, occurrences in counts.items():
            page_ratio = occurrences / total_pages
            if page_ratio >= min_page_ratio:
                patterns.append(BoilerplatePattern(
                    normalized_text=normalized_text,
                    occurrences=occurrences,
                    page_ratio=page_ratio,
                    position=position,
                ))
    return patterns

The min_page_ratio threshold is the decision that matters most. The rule isn't "delete any line that appears twice." It's "treat a structurally similar line as boilerplate if it keeps showing up in a page-margin position across most of the document." For a 100-page policy where CONFIDENTIAL - INTERNAL USE ONLY heads 98 pages, the evidence is strong.

One related point: taking something out of the text used for embedding doesn't mean deleting it from the system. Sometimes information should move from the semantic content layer to the metadata layer. Fields such as version, owner, classification and effective date are better kept as metadata than copied into each chunk.

Deduplication beyond exact copies

A document can be perfectly encoded, free of OCR damage and stripped of boilerplate and still hurt retrieval simply because the corpus holds it several times. Enterprise repositories are full of this:

AML_Policy_v8_Final.docx
AML_Policy_v8_Final_Approved.docx
AML_Policy_v8_Final_Copy.docx
AML_Policy_v8_Approved_2026.docx

Index all four files as separate documents and their chunks end up competing in retrieval. Ask about enhanced monitoring of high-risk customers and you could get three slots filled by one policy section, leaving two for truly different evidence. Duplicates cost more than disk space: they skew what top-k actually means.

Exact duplicates with content hashing

import hashlib
from dataclasses import dataclass

@dataclass
class DeduplicationRecord:
    document_id: str
    source: str
    content_hash: str
    duplicate_of: str | None = None
def canonicalize_for_hashing(text: str) -> str:
    """
    Deterministic normalization before hashing.
    Does not lowercase or remove punctuation because those transformations
    could collapse documents that are not actually identical.
    """
    lines = [" ".join(line.split()) for line in text.splitlines()]
    return "\n".join(lines).strip()
def calculate_content_hash(text: str) -> str:
    canonical_text = canonicalize_for_hashing(text)
    return hashlib.sha256(canonical_text.encode("utf-8")).hexdigest()
def find_exact_duplicates(
    documents: list[dict],
) -> tuple[list[dict], list[DeduplicationRecord]]:
    """
    Keep one canonical copy of each identical document while preserving
    duplicate lineage for auditability.
    """
    seen_hashes: dict[str, str] = {}
    unique_documents: list[dict] = []
    records: list[DeduplicationRecord] = []
    for document in documents:
        content_hash = calculate_content_hash(document["clean_text"])
        if content_hash in seen_hashes:
            records.append(DeduplicationRecord(
                document_id=document["document_id"],
                source=document["source"],
                content_hash=content_hash,
                duplicate_of=seen_hashes[content_hash],
            ))
            continue
        seen_hashes[content_hash] = document["document_id"]
        unique_documents.append(document)
        records.append(DeduplicationRecord(
            document_id=document["document_id"],
            source=document["source"],
            content_hash=content_hash,
        ))
    return unique_documents, records

The hash is also handy for incremental indexing. If the same normalized document turns up in tomorrow night's ingestion run, its hash is already known, so the pipeline can skip cleaning, chunking and embedding it again.

Near-duplicates with MinHash LSH

Exact hashing covers the easy case. The harder one is two versions of a KYC policy, say version 7 and version 8, that are 98% identical in text but differ on one regulatory clause. From a compliance viewpoint they're emphatically not duplicates, because that last clause changes the control requirement.

In a regulated RAG system, near-duplicate detection should first answer a different question, namely which documents are similar enough to be worth investigating, and leave deletion as a separate decision. For 200,000 documents, pairwise comparison needs about 20 billion comparisons. MinHash with locality-sensitive hashing avoids that by building compact signatures and placing similar ones into candidate buckets.

import re
from datasketch import MinHash, MinHashLSH

def create_word_shingles(text: str, shingle_size: int = 5) -> set[str]:
    """
    Five-word shingles work well for long policy and regulatory documents
    because they capture local textual structure without being overly
    sensitive to isolated formatting changes.
    """
    tokens = re.findall(r"\b\w+\b", text.casefold())
    if len(tokens) < shingle_size:
        return {" ".join(tokens)} if tokens else set()
    return {
        " ".join(tokens[i:i + shingle_size])
        for i in range(len(tokens) - shingle_size + 1)
    }
def create_minhash(text: str, num_perm: int = 128) -> MinHash:
    shingles = create_word_shingles(text)
    minhash = MinHash(num_perm=num_perm)
    for shingle in shingles:
        minhash.update(shingle.encode("utf-8"))
    return minhash
def build_duplicate_index(
    documents: list[dict],
    threshold: float = 0.85,
    num_perm: int = 128,
) -> tuple[MinHashLSH, dict[str, MinHash]]:
    """
    Build an LSH index for candidate near-duplicate discovery.
    The threshold identifies candidates. It does not automatically
    determine whether a document is deleted.
    """
    lsh = MinHashLSH(threshold=threshold, num_perm=num_perm)
    signatures: dict[str, MinHash] = {}
    for document in documents:
        document_id = document["document_id"]
        signature = create_minhash(document["clean_text"], num_perm=num_perm)
        signatures[document_id] = signature
        lsh.insert(document_id, signature)
    return lsh, signatures

When KYC policy v7 and v8 land in the same bucket, the pipeline has learned that they deserve a closer look, and nothing more. It hasn't learned that one should be removed. A production decision should combine text similarity with metadata such as policy ID, version, effective date, approval status and jurisdiction, because similarity alone can't draw that line.

OCR errors: right-looking text that is wrong

Encoding repair fixes characters that were stored the wrong way. OCR faults are a separate category. Here the engine may have read a perfectly valid Unicode character with full confidence, and simply picked the wrong one. Typical look-alike confusions are 0 and O, 1, l and I, 5 and S, and rn and m.

Overall error rates may be low, but aggregate accuracy isn't the metric that matters for regulated RAG. A 40-page circular that is 99.8% correct can still have its most critical values corrupted. The key architectural point is that OCR errors differ in risk. Fixing custorner to customer is low-risk, since context and a dictionary strongly suggest the intended word. Turning EUR l0,000 into EUR 10,000 belongs to another category entirely, because a wrong guess can alter a compliance obligation.

import re
from dataclasses import dataclass

@dataclass
class OCRIssue:
    token: str
    issue_type: str
    risk_level: str
    suggested_value: str | None
    requires_review: bool
CURRENCY_PATTERN = re.compile(r"\b(EUR|USD|GBP|INR)\s+([A-Za-z0-9.,]+)\b")
PERCENTAGE_PATTERN = re.compile(r"\b([A-Za-z0-9.,]+)\s*%")
def detect_numeric_ocr_issues(text: str) -> list[OCRIssue]:
    """
    Detect suspicious OCR substitutions inside financial values.
    This function identifies candidates. It does not silently rewrite
    high-risk values in the source text.
    """
    issues: list[OCRIssue] = []
    substitutions = {"O": "0", "o": "0", "I": "1", "l": "1"}
    def inspect_numeric_token(token: str, issue_type: str) -> None:
        if token.replace(",", "").replace(".", "").isdigit():
            return
        corrected = "".join(substitutions.get(char, char) for char in token)
        numeric_candidate = corrected.replace(",", "").replace(".", "")
        if numeric_candidate.isdigit() and corrected != token:
            issues.append(OCRIssue(
                token=token,
                issue_type=issue_type,
                risk_level="high",
                suggested_value=corrected,
                requires_review=True,
            ))
    for match in CURRENCY_PATTERN.finditer(text):
        inspect_numeric_token(match.group(2), issue_type="currency_value")
    for match in PERCENTAGE_PATTERN.finditer(text):
        inspect_numeric_token(match.group(1), issue_type="percentage")
    return issues

The function doesn't modify the document, deliberately. A regular expression can show that l0,000 looks suspicious in a currency context, but it can't prove the source actually said 10,000.

Ordinary vocabulary is better handled by a dictionary that knows the domain. Banking text needs a general word list plus terms such as beneficial ownership, counterparty, SWIFT, IBAN, SEPA and sanctioned. Otherwise a stock spell checker will happily "correct" legitimate financial terms into worse ones.

In production, the call on OCR fixes weighs engine confidence against how much a mistake would matter. Confident readings of low-stakes text can usually be repaired without a human. High-risk content, meaning monetary values, percentages, regulatory article numbers and dates, should go to review whatever the OCR confidence. A financial system should aim for controlled correctness and not maximum automation.

Multilingual preprocessing

Language detection seems minor until a corpus crosses borders. A payment-operations repository might contain English process documents, German customer letters, French regulatory notices and Spanish contracts, and one AML investigation file can hold several languages in a single case record.

Language affects everything downstream. Spell correction depends on it, OCR error patterns vary by script, named entities carry accents that must survive normalization, and the embedding model itself may behave differently depending on the language indexed.

from dataclasses import dataclass
from langdetect import DetectorFactory, LangDetectException, detect_langs

DetectorFactory.seed = 0  # Makes pipeline output reproducible.
@dataclass
class LanguagePrediction:
    language: str
    confidence: float
    alternatives: list[tuple[str, float]]
    needs_review: bool
def detect_document_language(
    text: str,
    confidence_threshold: float = 0.85,
) -> LanguagePrediction:
    sample = text.strip()
    if not sample:
        return LanguagePrediction(language="unknown", confidence=0.0, alternatives=[], needs_review=True)
    try:
        predictions = detect_langs(sample)
    except LangDetectException:
        return LanguagePrediction(language="unknown", confidence=0.0, alternatives=[], needs_review=True)
    best = predictions[0]
    alternatives = [(p.lang, round(p.prob, 4)) for p in predictions]
    return LanguagePrediction(
        language=best.lang,
        confidence=best.prob,
        alternatives=alternatives,
        needs_review=best.prob < confidence_threshold,
    )

When a document mixes languages, detecting at paragraph level beats one verdict for the whole file. It also hands the chunker a hint about structure: if a split lands across a language boundary the splitter never saw, a single chunk can end up with two unrelated linguistic contexts.

Some teams translate everything into English first. That makes the index simpler, yet it inserts one more transformation between the authoritative text and what gets retrieved, and a German compliance clause may use wording with a precise legal sense that a translation subtly shifts. A sturdier design treats the original-language text as the source of truth and stores any translation as an extra representation, with provenance recorded in metadata. Language metadata should be precise too:

metadata = {
    "primary_language": "en",
    "languages": ["en", "de", "fr"],
    "language_distribution": {"en": 0.72, "de": 0.21, "fr": 0.07},
    "is_multilingual": True,
}

Language detection can also act as a health check. Suppose a UK AML policy, expected in English, suddenly reports primary_language = "unknown" at confidence = 0.31. The file may not be multilingual; OCR could have mangled the text so badly that the detector no longer recognizes it.

PII and sensitive data: detect before embedding

Preprocessing is one of the last chances to control what enters the retrieval infrastructure. A KYC file may hold customer names, passport numbers, IBANs, phone numbers and email addresses. After those values pass through an embedding service and into a vector database, the problem is much harder than securing the original document, because copies now exist in embedding requests, vectors, metadata, caches, indexes, logs and backups.

The question isn't whether the embedding model can understand PII, since it can. It's whether the information needs to be embedded at all.

Deleting every entity the detector finds does make the corpus safer, but it ruins retrieval by entity. If each company name is replaced by <ORGANIZATION>, an AML investigator asking about Northbridge Trading GmbH finds nothing. So the production route is to give every detected entity type a handling policy before acting. Public and operational names are generally kept, customer PII is redacted, tokenized or restricted, internal organizational references follow access rules, and credentials never go near the index.

from dataclasses import dataclass, field
from presidio_analyzer import AnalyzerEngine

@dataclass
class SensitiveEntity:
    entity_type: str
    start: int
    end: int
    score: float
    original_value: str
@dataclass
class PIIScanResult:
    entities: list[SensitiveEntity] = field(default_factory=list)
    needs_review: bool = False
class BankingPIIDetector:
    def __init__(self) -> None:
        self.analyzer = AnalyzerEngine()
    def detect(
        self,
        text: str,
        language: str = "en",
        score_threshold: float = 0.70,
    ) -> PIIScanResult:
        """
        Detect sensitive entities without modifying the source text.
        Detection is deliberately separated from anonymization because
        different entity types require different handling policies.
        """
        results = self.analyzer.analyze(
            text=text, language=language, score_threshold=score_threshold
        )
        entities = [
            SensitiveEntity(
                entity_type=r.entity_type,
                start=r.start,
                end=r.end,
                score=r.score,
                original_value=text[r.start:r.end],
            )
            for r in results
        ]
        high_risk_types = {"CREDIT_CARD", "IBAN_CODE", "US_SSN", "IP_ADDRESS"}
        return PIIScanResult(
            entities=entities,
            needs_review=any(e.entity_type in high_risk_types for e in entities),
        )

Replacing values with typed placeholders usually works better for RAG than wiping them:

Customer IBAN: <IBAN>
Customer Email: <EMAIL_ADDRESS>

The sentence keeps its meaning-bearing shape, so a question about where a customer's IBAN gets verified can still match text containing <IBAN>. When relationships between mentions must survive, deterministic tokens beat generic placeholders. CUSTOMER_7F29A1 recurring through 40 case records keeps an investigative trail intact, while swapping every name for <PERSON> would cut it.

Metadata is an easily forgotten leak. A body text can be redacted with care and still ship customer_name: "Lena Hoffmann", or a source path such as /customers/Lena_Hoffmann/KYC/passport_review.pdf, in its fields. Sensitive-data controls therefore need to cover everything that gets indexed, not only the text.

A production-grade pipeline

Individual cleaning functions are enough while you're experimenting. In production a stricter standard applies: a document must not move on to chunking just because a chain of steps ran without an exception. The pipeline has to record what it altered, what still looks suspicious, whether sensitive data turned up, whether the file duplicates another, and whether the end result deserves to be indexed.

from dataclasses import dataclass, field
from enum import Enum
from typing import Any

class QualityStatus(str, Enum):
    PASS = "pass"
    REVIEW = "review"
    REJECT = "reject"
@dataclass
class Transformation:
    stage: str
    operation: str
    details: dict[str, Any] = field(default_factory=dict)
@dataclass
class QualityFlag:
    code: str
    severity: str
    details: dict[str, Any] = field(default_factory=dict)
@dataclass
class RetrievalReadyDocument:
    document_id: str
    source: str
    raw_text: str
    clean_text: str
    language: str | None = None
    language_confidence: float | None = None
    content_hash: str | None = None
    duplicate_of: str | None = None
    transformations: list[Transformation] = field(default_factory=list)
    quality_flags: list[QualityFlag] = field(default_factory=list)
    metadata: dict[str, Any] = field(default_factory=dict)
    status: QualityStatus = QualityStatus.PASS

The difference between raw_text and clean_text is fundamental. The first answers what ingestion actually extracted, and the second answers which representation we're willing to chunk and embed. A transformation log explains how one became the other, and that lineage is invaluable when a retrieval failure is reported six months later.

import logging
logger = logging.getLogger(__name__)

class ProductionDocumentCleaner:
    def __init__(
        self,
        pii_detector: BankingPIIDetector,
        pii_anonymizer,
        redact_pii_for_embeddings: bool = True,
    ) -> None:
        self.pii_detector = pii_detector
        self.pii_anonymizer = pii_anonymizer
        self.redact_pii_for_embeddings = redact_pii_for_embeddings
    def clean(self, document: dict) -> RetrievalReadyDocument:
        result = RetrievalReadyDocument(
            document_id=document["document_id"],
            source=document["source"],
            raw_text=document["text"],
            clean_text=document["text"],
            metadata=document.get("metadata", {}).copy(),
        )
        try:
            self._normalize_text(result)
            self._detect_language(result)
            self._validate_ocr(result)
            self._handle_sensitive_data(result)
            self._calculate_hash(result)
            self._apply_quality_gate(result)
        except Exception as exc:
            logger.exception("Preprocessing failed for document %s", result.document_id)
            result.quality_flags.append(QualityFlag(
                code="PREPROCESSING_EXCEPTION",
                severity="critical",
                details={"error": str(exc)},
            ))
            result.status = QualityStatus.REJECT
        return result
    def _apply_quality_gate(self, document: RetrievalReadyDocument) -> None:
        severities = {flag.severity for flag in document.quality_flags}
        if "critical" in severities:
            document.status = QualityStatus.REJECT
        elif "high" in severities:
            document.status = QualityStatus.REVIEW
        else:
            document.status = QualityStatus.PASS

The gate outputs one of three explicit states. PASS lets the document continue to chunking with no unresolved high-risk findings. REVIEW means usable content came out, but certain transformations need a person's validation before indexing. REJECT means the result can't be trusted enough to index. A scanned circular with a doubtful l0% threshold goes to REVIEW; a PDF that produced just 17 garbled characters across 80 pages probably lands in REJECT; and a policy whose only fixes were apostrophes and whitespace is a PASS.

Metrics for the cleaning stage are worth keeping as well. If text_reduction_ratio normally trims 4-8% of a document and jumps to 63% after a release, something has gone wrong. If unknown_language_rate leaps from 0.3% to 18% following an OCR library upgrade, the detector is revealing a regression further upstream. Numbers like these make cleaning visible and manageable instead of an unseen script.

Trade-offs in production

No cleaning configuration is right for everyone, and the choices carry real consequences.

• Encoding repair. ftfy makes a cheap and effective first pass over typical mojibake and Unicode damage. Treat it as a partner to deterministic Unicode normalization, not a licence to rewrite text at will.
• Deduplication. SHA-256 hashing is nearly free and should deal with exact duplicates before pricier similarity techniques. MinHash LSH is useful for finding near-duplicates at scale, but its threshold should produce candidates, not deletion decisions.
• OCR correction. Dictionary lookup costs little and behaves predictably on everyday words. An LLM that reads context can catch harder cases, but a generative model in preprocessing brings extra cost plus a fresh risk, plausible-sounding rewrites. For regulated material, let it propose fixes in a structured format that a reviewer approves rather than letting it edit documents unseen.
• Multilingual handling. Whole-document detection is inexpensive and fine for routing. Paragraph-level detection fits mixed-language files better at a higher cost; which one wins depends on your real language distribution and on retrieval benchmarks. The word "multilingual" on a model card says little about retrieving German AML terms from an English compliance question.
• PII. Strong redaction lowers exposure but kills lookups by entity. Often the better design keeps semantic vector search separate from a secure exact-match lookup for identifiers, so one index isn't asked to solve both problems.

The thread running through all of these: let the retrieval requirements and the danger of distorting meaning drive preprocessing decisions, and ignore how neat the cleaned text looks.

Back to the opening policy

Return to that original KYC policy. Ingestion extracted it successfully, and the cleaning pipeline now does a great deal more before letting it into the index. It repairs broken encoding without stripping legitimate multilingual characters, removes repeated page headers while keeping document metadata, spots exact copies stored across repositories, and treats older policy versions as related and not as items to delete blindly. It marks EUR l0,000 as a serious OCR anomaly and refuses to guess quietly, recognizes multilingual sections, detects sensitive customer identifiers before they reach the embedding layer, records every transformation, and decides whether the result should pass, go to review or be rejected.

Only then does the document reach chunking. The order matters because chunking can't recover information that preprocessing already destroyed, and embeddings can't tell authoritative knowledge from corruption that was let into the corpus unchecked.

What comes after cleaning is a different puzzle: where do you cut a clean 80-page AML policy? A plain 500-token window is trivial to build, yet it can separate a rule from its exception, strip a table from its heading, or glue unrelated clauses together merely because they share a token budget. Splitting approaches range from fixed-size and recursive through semantic, structure-aware, parent-child and proposition-level, and each deserves separate treatment.

A checklist before chunking

Before a file moves on to chunking and embedding, the preprocessing stage ought to be able to give a clear answer to each of the following:

• Has encoding and Unicode normalization run without dropping characters that carry meaning?
• Were control characters, zero-width characters, repeated headers and footers, and other retrieval noise handled?
• Were exact and near duplicates detected, with genuine document versions left intact?
• Are dubious OCR readings of amounts, percentages, dates, identifiers and legal references reported to someone, not quietly fixed?
• Was the language detected at a suitable granularity, including mixed-language content?
• Was sensitive information classified under the application's security policy before embedding?
• Could each significant change be followed back to the originally parsed text?
• Is there an explicit PASS, REVIEW or REJECT verdict before chunking begins?

If any answer is unclear, the document isn't retrieval-ready.

If a damaged policy has already been split into 80 chunks, embedded, copied across a vector index, cached and served by a live application, repairing the source file no longer suffices. Every derived chunk must be located, its vectors deleted, caches invalidated, the document reprocessed, and the removal of stale representations verified. Rejecting bad knowledge is cheapest before it turns into an embedding.

The code shown aims for clarity and correctness. For production, add structured logging, configuration and secrets management, persistent audit trails for transformations, preprocessing metrics, failure recovery and human review for high-risk corrections. In regulated settings every destructive transformation should be traceable to the original source.