Choosing and Tuning Embedding Models for Production RAG Systems

This is the fifth installment in a series on building production-grade retrieval-augmented generation systems, following the path from raw documents to a system that can answer real questions. Earlier stages in this series covered cleaning and normalizing extracted content, then splitting that content into retrieval units through chunking. With chunks in hand, the next job is turning them into something a search index can actually compare.

Picture the same relationship manager introduced earlier, still stuck on a EUR 12 million credit facility for a high-risk corporate client. Answering the request means locating an Enhanced Due Diligence requirement buried in a policy clause, an approval threshold sitting in a table row, and a control sequence described in a process diagram. The chunking stage has already carved these out as separate, traceable pieces of content.

Even so, none of that is searchable yet by a vector index.

An embedding model is what turns each chunk into a fixed-length numeric vector. The same model then converts an incoming query into a vector, and the index retrieves whichever chunks land nearest to it in that vector space. Whether a phrase like "Group Credit Committee approval" in a policy document and "GCC sign-off threshold" in a user's question actually end up near each other is not guaranteed — it hinges entirely on which model you picked and what it learned during training.

That dependency is what this part of the series is about.

Choosing an embedding model amounts to betting on how much vocabulary and phrasing your documents share with your users' queries. Pick the wrong one for your domain, and you'll burn weeks chasing what looks like a retrieval bug but is actually a representation problem — the vectors themselves are placed wrong, so no amount of index tuning will fix it.

What an Embedding Model Actually Computes

At its core, an embedding model ingests a token sequence and outputs one dense vector, usually somewhere between 384 and 3072 dimensions depending on the specific model. That vector is meant to be a compressed encoding of what the input means.

The working assumption behind retrieval is that inputs with similar meaning end up with vectors that are close together in that space. Closeness is most commonly measured with cosine similarity, which looks at the angle separating two vectors rather than their raw distance — a property that makes it insensitive to differences in text length.

Consider a compliance rule stating that corporate clients classified as high-risk are required to go through Enhanced Due Diligence checks before a credit proposal can be submitted for review. A capable general-purpose model would likely place its vector close to comparable compliance wording from other banks, near legal material about due-diligence duties, and near regulatory guidance on managing high-risk clients.

A relationship manager, though, might phrase the same underlying need very differently, asking something like what steps come before a credit request can go in. Whether that everyday phrasing lands near the formal policy vector is really a question of how much training data paired informal operational questions with formal compliance language. Models trained mostly on broad, general-purpose web text often never absorb that specific pairing when the field is specialized.

This gap between everyday query wording and specialized document wording is known as vocabulary mismatch, and it is the leading cause of retrieval quality problems in enterprise RAG systems.

Tokenisation and the Context Window

Before an embedding model computes anything, it first breaks the input into tokens using its own internal vocabulary. Token counts don't map cleanly onto word counts or character counts. A chunk of 500 tokens in English might correspond to roughly 350–400 words, whereas the same token budget in German — where words are frequently compounded — could represent fewer distinct ideas.

Every embedding model imposes a maximum context length, and anything longer either gets truncated or needs special handling. Sentence-transformers models typically cap out somewhere between 256 and 512 tokens. OpenAI's text-embedding-3-large handles up to 8,191 tokens. BGE-M3 goes as high as 8,192 tokens. Jina embeddings v3 also supports up to 8,192 tokens.

The practical takeaway for production RAG is that the chunk boundaries decided earlier in the pipeline must stay within whatever context limit the chosen embedding model imposes. Any chunk that overshoots that limit gets silently cut off, and the resulting vector only reflects a portion of the original text — a failure mode that won't show up anywhere in your pipeline's logs.

The Semantic Space and When It Breaks

Most current embedding models are transformer encoders trained with a contrastive objective: pairs of texts that mean similar things are drawn together in vector space, while dissimilar pairs are pushed apart. After enough training, the model ends up with a geometric layout where proximity is a proxy for semantic relatedness.

That layout performs well as long as queries and documents share terminology, writing style, and conceptual framing with whatever the model was trained on. For enterprise RAG, it tends to fail in a few predictable ways:

• Domain-specific terminology produces failures that are easy to miss. An analyst asking about the "SAR filing threshold for structuring" may get no match against a policy chunk phrased as "Suspicious Activity Report submission criteria for transaction structuring," if the model never learned that these two phrasings mean the same thing.
• Abbreviations don't behave consistently. Terms like "GCC" (Group Credit Committee), "EDD" (Enhanced Due Diligence), and "RFI" (Request for Information) can get embedded by a general-purpose model according to their more common meanings elsewhere — Gulf Cooperation Council, Electronic Document Delivery, Radio Frequency Interference.
• Regulatory identifiers hold no inherent meaning for general models. A code like CRD-EU-047, run through a general-purpose embedding model, is treated as an arbitrary string. A model trained specifically on regulatory text would instead place it alongside other EU credit-regulation identifiers that belong to the same conceptual family.
• Numeric thresholds are handled only partially well by general models. The phrase "EUR 10 million" on its own gets embedded near other monetary figures. When it appears alongside language about approval authority, a model fine-tuned on the domain can capture the link between that specific amount and the governance control it triggers — something a general model is less likely to encode correctly.

Recognizing exactly where general-purpose embeddings tend to break down matters just as much as knowing which model tops a public benchmark leaderboard.

Choosing an Embedding Model in 2025

The field of embedding models has narrowed considerably. The comparison that follows covers the models that matter most for enterprise RAG systems in banking and financial services as of mid-2025.

There is no universal winner across every enterprise scenario. Your choice hinges on the mix of languages you need to support, the latency budget and infrastructure you have available, whether local inference is even an option for you, and how large the terminology gap is between your domain and what general-purpose models were trained on — a gap that determines whether fine-tuning is worth the effort.

Asymmetric Retrieval and Telling the Model What Kind of Input It's Getting

One distinction many teams overlook is asymmetric retrieval. When you retrieve passages, the query and the chunk you're matching it against are structurally very different things. Queries tend to be short, phrased as questions, and often missing much of the vocabulary that appears in a correct answer. Passages, by contrast, are longer, stated as facts, and dense with domain terms.

Certain models are built to recognize this asymmetry directly. E5 models add a "query:" or "passage:" prefix to the input text so the model knows which role it's embedding. Cohere's Embed v3 exposes this through an input_type parameter, with accepted values including "search_query", "search_document", "classification", and "clustering".

Supplying the wrong input type at indexing or query time quietly hurts your similarity scores in ways that are hard to trace back to a root cause. If you embed a document as though it were a query, you get a vector shaped for query geometry rather than passage geometry. Retrieval doesn't fail outright — it just loses precision in a way that's easy to miss during casual testing.

In production, don't rely on developers remembering to set this correctly — enforce it through configuration. Both the indexing-time embedding call and the query-time embedding call should explicitly declare their input type whenever the model you're using supports that option.

import cohere
from typing import List

co = cohere.Client(api_key="your_api_key")

def embed_documents(chunks: List[str]) -> List[List[float]]:
    """Embed document chunks for indexing with explicit document input type."""
    response = co.embed(
        texts=chunks,
        model="embed-english-v3.0",
        input_type="search_document",
        embedding_types=["float"]
    )
    return response.embeddings.float

def embed_query(query: str) -> List[float]:
    """Embed a search query with explicit query input type."""
    response = co.embed(
        texts=[query],
        model="embed-english-v3.0",
        input_type="search_query",
        embedding_types=["float"]
    )
    return response.embeddings.float[0]

Sparse Vectors: Where Keyword Matching Beats Semantic Search

Dense embeddings capture meaning. Sparse representations instead capture which terms are present and how heavily they should be weighted. For a meaningful share of query types in enterprise RAG, sparse retrieval outperforms dense retrieval outright, and in most production systems, blending the two beats using either one alone.

BM25 as the Reliable Baseline

BM25 remains the standard approach to keyword-driven retrieval. It scores relevance using how often a term appears in a document, how rare that term is across the whole corpus, and a normalisation factor that accounts for document length. There's no model to train, no GPU requirement, and no embedding API call involved.

Consider a query such as "CRD-EU-047 approval authority threshold." BM25 will rank highly any chunk containing those literal terms. A dense model, on the other hand, might not surface that chunk unless its training corpus happened to build a strong association between that specific policy code and the concept of approval authority.

from rank_bm25 import BM25Okapi
import re
from typing import List, Tuple

def tokenise(text: str) -> List[str]:
    """Simple whitespace and punctuation tokeniser for BM25."""
    return re.findall(r'\b\w+\b', text.lower())

class BM25Index:
    def __init__(self, documents: List[str]):
        self.documents = documents
        tokenised = [tokenise(doc) for doc in documents]
        self.bm25 = BM25Okapi(tokenised)

    def search(self, query: str, top_k: int = 10) -> List[Tuple[int, float]]:
        """Return (doc_index, score) pairs for the top_k results."""
        tokens = tokenise(query)
        scores = self.bm25.get_scores(tokens)
        ranked = sorted(enumerate(scores), key=lambda x: x[1], reverse=True)
        return ranked[:top_k]

Dense retrieval is good at capturing conceptual similarity, while sparse retrieval is good at catching exact matches on terms and identifiers. Combining both — hybrid retrieval — tends to pay off especially well for banking corpora, where regulatory language is precise and identifier-heavy.

In corpora built on stable, precisely defined policy language, BM25 alone often matches dense retrieval's recall on narrow factual lookups, at a fraction of the infrastructure overhead. Its main blind spot is synonymy: a query using "EDD requirements" won't retrieve a passage that only says "Enhanced Due Diligence requirements" unless the exact phrase overlaps somewhere.

SPLADE: Sparse Vectors That Learn Vocabulary Expansion

SPLADE (Sparse Lexical and Expansion Model) occupies the middle ground between plain keyword matching and full dense retrieval. At indexing time, a masked language model is used to enrich both documents and queries with semantically related vocabulary that isn't necessarily present in the original wording. The result is a sparse vector whose dimensions each correspond to a specific vocabulary token, weighted according to how important that token is to the input.

So a SPLADE-encoded passage discussing EDD requirements might carry expanded weight on tokens like "customer due diligence," "risk assessment," and "identity verification," even when none of those exact phrases appear in the source text. This expansion boosts recall on queries that rely on synonyms, while still preserving the efficiency and interpretability that come with sparse, inverted-index-friendly representations.

The trade-off is added inference cost at index time and a larger storage footprint than plain BM25. But for financial-services corpora where the same policy concept gets described differently across jurisdictions and document revisions, that expansion can noticeably widen retrieval coverage.

Matryoshka Embeddings: Adjustable Vector Size for Cost Control

Matryoshka Representation Learning (MRL) builds embeddings where the leading N dimensions already form a complete, self-contained representation of the input — additional dimensions layer on finer detail rather than overwrite what came before.

The technique borrows its name from Russian nesting dolls: a 1536-dimension Matryoshka vector contains a fully functional 256-dimension representation within its first 256 slots, a functional 512-dimension representation within its first 512 slots, and so on up the chain.

OpenAI's text-embedding-3 model family supports this directly via a dimensions parameter.

from openai import OpenAI
from typing import List

client = OpenAI()

def embed_with_matryoshka(
    texts: List[str],
    dimensions: int = 256,
    model: str = "text-embedding-3-large"
) -> List[List[float]]:
    """
    Embed texts at a specified sub-dimension.
    Lower dimensions reduce storage and index cost.
    Measure retrieval quality drop before committing to a dimension.
    """
    response = client.embeddings.create(
        input=texts,
        model=model,
        dimensions=dimensions
    )
    return [item.embedding for item in response.data]

Matryoshka embeddings nest increasingly detailed representations inside one vector: the first 256 dimensions already give you a workable retrieval representation, and each additional tier adds semantic precision at a proportional cost in storage.

The real production benefit is being able to adjust the storage-versus-quality trade-off on demand, without retraining a model or rebuilding your index from scratch. For a banking policy corpus, you could benchmark retrieval at 256, 512, 1024, and 3072 dimensions, and discover that 512 dimensions delivers 97% of full recall while using only 17% of the storage that the full vector would require.

That trade-off is rarely as tidy in practice as it looks in a benchmark chart. Fine domain distinctions — especially between closely related regulatory concepts — often live specifically in the higher-dimensional part of the vector. Test against your own corpus and real query patterns before locking in a reduced dimensionality for production use.

Compressing Embeddings Without Sacrificing Much Accuracy

Standard embeddings store every dimension as a 32-bit float. Scale that up to a million document chunks at 1536 dimensions each, and you are looking at roughly 6 GB of raw vectors before any indexing overhead is added. At enterprise scale, that storage footprint and its associated memory cost stop being a rounding error.

Quantisation addresses this by reducing the number of bits used to represent each dimension. Three techniques dominate in practice: scalar quantisation (converting float32 to int8), binary quantisation (converting float32 to a single bit), and product quantisation (compressing each vector into a shorter code).

Scalar Quantisation: int8

Scalar quantisation maps the continuous float32 range onto 256 discrete integer values. Each dimension shrinks from 4 bytes to 1, cutting storage by 75%. Because high-dimensional embedding models spread information thinly across many dimensions, no single dimension carries much weight on its own, so the accuracy lost to this rounding is usually minor.

import numpy as np
from typing import Tuple

def quantise_to_int8(
    embeddings: np.ndarray
) -> Tuple[np.ndarray, float, float]:
    """
    Scalar quantisation to int8.
    Returns quantised array plus the scale and zero_point needed for dequantisation.
    """
    min_val = embeddings.min()
    max_val = embeddings.max()
    scale = (max_val - min_val) / 255.0
    zero_point = -round(min_val / scale)
    quantised = np.clip(
        np.round(embeddings / scale) + zero_point,
        0, 255
    ).astype(np.uint8)
    return quantised, scale, zero_point

def dequantise_from_int8(
    quantised: np.ndarray,
    scale: float,
    zero_point: float
) -> np.ndarray:
    """Reconstruct approximate float32 embeddings from int8."""
    return ((quantised.astype(np.float32) - zero_point) * scale)

Binary Quantisation

Binary quantisation goes further, collapsing each dimension to a single bit that simply records whether the original float value was positive or negative. This cuts storage by roughly 97% compared to float32. Because the representation is no longer continuous, similarity is measured with Hamming distance instead of cosine similarity.

This technique works best on models whose output distributions are naturally well-centred, so that for any given input roughly half the dimensions land on each side of zero. If a model's dimensions are skewed rather than balanced, binary quantisation causes noticeably more quality loss. Cohere built Embed v3 with this constraint in mind, and Anthropic's published evaluation of that model reports under 1% retrieval degradation alongside a 97% storage reduction on their test sets. Treat that figure as a starting point rather than a guarantee, and validate it against your own corpus before depending on it.

import numpy as np

def quantise_to_binary(embeddings: np.ndarray) -> np.ndarray:
    """
    Binary quantisation: positive dimensions become 1, negative become 0.
    Packs 8 dimensions per byte using numpy packbits.
    """
    binary_matrix = (embeddings > 0).astype(np.uint8)
    return np.packbits(binary_matrix, axis=1)

def hamming_similarity(
    query_binary: np.ndarray,
    corpus_binary: np.ndarray
) -> np.ndarray:
    """Compute normalised Hamming similarity for binary embeddings."""
    n_bits = corpus_binary.shape[1] * 8
    xor = np.bitwise_xor(
        query_binary,
        corpus_binary
    )
    hamming_distances = np.unpackbits(xor, axis=1).sum(axis=1)
    return 1.0 - (hamming_distances / n_bits)

In production, binary quantisation is usually deployed as the first stage of a two-step retrieval pipeline: a binary index handles fast, broad candidate recall, and a full-precision pass then re-scores the top results. This setup keeps most of the storage saving while restoring precision where it matters, at the final ranking step.

Fine-Tuning for Domain-Specific RAG

Fine-tuning is the correct response once you've confirmed that general-purpose embeddings genuinely fall short on your data. The objective is to teach the model that the vocabulary, shorthand, and conceptual links specific to your domain belong close together in semantic space.

Fine-tuning isn't always warranted, and it isn't always the right fix. If your retrieval problems trace back to poor chunking, as discussed earlier in this series, adjusting the embedding model won't help. If the root cause is how reranking is configured or how prompts are built, fine-tuning targets the wrong layer of the system entirely. Before committing resources to it, break down your retrieval failures by query type to confirm where the actual problem sits.

When General Embeddings Fail

In banking RAG specifically, a handful of recurring failure patterns justify the investment in fine-tuning:

Domain-specific abbreviations get misread. A general-purpose model might link "NPA" to the National Parks Association instead of Non-Performing Asset, and it may only weakly connect "KYC" to the compliance and onboarding concepts that actually dominate banking queries.

Connections between related concepts across documents go missing. A search for "facility restructuring provisions" should surface policy text about "loan modification frameworks," but a model trained mostly on general web content may never have seen these phrasings paired closely enough to build that bridge.

Regulatory codes and identifiers don't get enough weight. Policy version numbers, regulation codes, and jurisdiction markers should meaningfully shape ranking, yet general embedding models tend to treat them as low-value tokens carrying little semantic signal.

Numeric thresholds lose their governance context. A figure like "EUR 10 million" embedded on its own shouldn't automatically match a query about "approval authority for large exposures" — that link only forms if the model has been trained on domain data that ties the number to its regulatory meaning.

Building Training Pairs from Domain-Specific Data

Fine-tuning sentence-transformers models under a contrastive objective depends on positive pairs: examples pairing a query with a passage the model should learn to treat as related. Negative examples can either be hand-picked or drawn automatically from the surrounding corpus.

In a banking RAG context, these positive pairs can be assembled from a handful of practical sources:

Existing question-answer sets already produced by compliance and credit teams, where each question is linked to its source passage.

The natural structure of policy documents, where a heading paired with the paragraph beneath it forms a ready-made positive example.

Logged analyst queries paired with the passages that were actually retrieved when the answer was correct.

Machine-generated queries produced by an LLM for each chunk of text, using that chunk itself as the matching positive passage.

Of these, generating synthetic queries tends to be the most workable path when little labeled data already exists.

from openai import OpenAI
import json
from typing import List, Dict

client = OpenAI()

def generate_training_queries(
    chunk: str,
    chunk_metadata: Dict,
    n_queries: int = 3
) -> List[Dict]:
    """
    Generate synthetic query-passage pairs for fine-tuning.
    The chunk itself is the positive passage for each generated query.
    """
    prompt = f"""You are generating training data for a banking RAG system.
Given the following policy passage, generate {n_queries} realistic questions
that a credit analyst, compliance officer, or relationship manager might ask
that this passage directly answers. Each question should use natural language
and may use different terminology than the passage itself.

Passage:
{chunk}

Return a JSON array of objects with keys "query" and "difficulty".
Difficulty should be "narrow" (single fact) or "synthesis" (multiple facts).
Return only the JSON array, no other text."""

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"}
    )

    try:
        result = json.loads(response.choices[0].message.content)
        queries = result.get("queries", result) if isinstance(result, dict) else result
        return [
            {
                "query": q["query"],
                "passage": chunk,
                "document_id": chunk_metadata.get("document_id"),
                "chunk_id": chunk_metadata.get("chunk_id"),
                "difficulty": q.get("difficulty", "narrow")
            }
            for q in queries
        ]
    except (json.JSONDecodeError, KeyError):
        return []

Contrastive Training with Triplet-Style Loss

The strongest training objective for retrieval-oriented embedding models is contrastive learning, using either in-batch negatives or deliberately chosen hard negatives. Sentence-transformers supports this pattern through MultipleNegativesRankingLoss, which uses every other example in a training batch as an implicit negative for a given anchor-positive pair.

from sentence_transformers import SentenceTransformer, InputExample
from sentence_transformers.losses import MultipleNegativesRankingLoss
from torch.utils.data import DataLoader
from typing import List, Dict
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

def build_training_examples(
    pairs: List[Dict]
) -> List[InputExample]:
    """
    Convert query-passage pairs into InputExample objects.
    MultipleNegativesRankingLoss expects (anchor, positive) pairs.
    Negatives are sampled automatically from other items in the batch.
    """
    return [
        InputExample(texts=[pair["query"], pair["passage"]])
        for pair in pairs
        if pair.get("query") and pair.get("passage")
    ]

def fine_tune_embedding_model(
    base_model_name: str,
    training_pairs: List[Dict],
    output_path: str,
    epochs: int = 3,
    batch_size: int = 16,
    warmup_steps: int = 100
) -> SentenceTransformer:
    """
    Fine-tune a sentence-transformers model on domain-specific query-passage pairs.

    base_model_name: HuggingFace model identifier or local path.
    training_pairs: List of dicts with "query" and "passage" keys.
    output_path: Directory to save the fine-tuned model.
    """
    model = SentenceTransformer(base_model_name)
    logger.info(f"Loaded base model: {base_model_name}")
    logger.info(f"Training on {len(training_pairs)} query-passage pairs")

    examples = build_training_examples(training_pairs)
    loader = DataLoader(examples, shuffle=True, batch_size=batch_size)
    loss = MultipleNegativesRankingLoss(model)

    total_steps = len(loader) * epochs
    logger.info(f"Training for {epochs} epochs, {total_steps} total steps")

    model.fit(
        train_objectives=[(loader, loss)],
        epochs=epochs,
        warmup_steps=warmup_steps,
        output_path=output_path,
        show_progress_bar=True,
        checkpoint_path=output_path,
        checkpoint_save_steps=len(loader)
    )

    logger.info(f"Fine-tuned model saved to: {output_path}")
    return model

Benchmark Results: A Banking Corpus Before and After Fine-Tuning

Consider a retrieval benchmark run against a corporate credit policy corpus made up of 847 chunks pulled from policy documents, approval matrices, enhanced due diligence procedures, and AML guidance. The evaluation set consists of 120 queries spanning four categories: narrow factual lookups, threshold questions, multi-evidence questions, and synthesis questions.

The gain from domain fine-tuning is most pronounced on threshold-style queries, such as asking for the approval threshold applying to high-risk corporate customers in the EU. This is where banking abbreviations and regulatory vocabulary diverge most sharply from what a general-purpose model has seen during pretraining, so closing that vocabulary gap yields the biggest jump in recall. Multi-evidence and synthesis queries improve less from fine-tuning on its own, but they gain considerably once hybrid retrieval is layered on top.

Treat these numbers as illustrative of what a well-executed fine-tuning project on a properly labeled banking dataset can deliver, not as a promise. Your own corpus composition, query mix, and labeling accuracy will shift the results. What matters most is breaking retrieval quality down by query type instead of reporting one blended score, since each query type tends to fail for different reasons.

Throughput Considerations for Batch Embedding at Scale

Feeding 100,000 chunks through an embedding API one request at a time is both slow and costly. Production-grade embedding pipelines instead batch their inputs so they can push throughput higher, stay within rate limits, recover cleanly from failures, and keep output generation deterministic.

Batching Through Provider APIs

OpenAI's embeddings endpoint allows up to 2,048 inputs in a single call. Cohere's Embed endpoint tops out at 96 texts per request unless you use their dedicated batch API for larger jobs. Running inference locally with sentence-transformers gives you configurable batch sizes, limited only by available GPU memory.

import time
import logging
from typing import List, Optional
from openai import OpenAI, RateLimitError, APIError

logger = logging.getLogger(__name__)
client = OpenAI()

def embed_in_batches(
    texts: List[str],
    model: str = "text-embedding-3-large",
    batch_size: int = 512,
    max_retries: int = 3,
    retry_delay: float = 2.0,
    dimensions: Optional[int] = None
) -> List[List[float]]:
    """
    Embed a large list of texts using batched API calls with retry logic.

    texts: Pre-chunked text strings. Caller is responsible for ensuring
           no text exceeds the model's token limit.
    batch_size: Number of texts per API call. Stay well below the API limit
                to avoid hitting per-request token limits.
    dimensions: Optional Matryoshka dimension reduction for supported models.
    """
    all_embeddings: List[List[float]] = []
    total_batches = (len(texts) + batch_size - 1) // batch_size

    for batch_idx in range(0, len(texts), batch_size):
        batch = texts[batch_idx: batch_idx + batch_size]
        current_batch = batch_idx // batch_size + 1
        logger.info(f"Embedding batch {current_batch}/{total_batches} "
                    f"({len(batch)} texts)")

        kwargs = {
            "input": batch,
            "model": model
        }
        if dimensions is not None:
            kwargs["dimensions"] = dimensions

        attempt = 0
        while attempt < max_retries:
            try:
                response = client.embeddings.create(**kwargs)
                # Preserve input order: API returns items sorted by index
                sorted_items = sorted(response.data, key=lambda x: x.index)
                all_embeddings.extend([item.embedding for item in sorted_items])
                break

            except RateLimitError:
                attempt += 1
                wait = retry_delay * (2 ** attempt)
                logger.warning(f"Rate limit hit on batch {current_batch}. "
                               f"Waiting {wait:.1f}s before retry {attempt}/{max_retries}")
                time.sleep(wait)

            except APIError as e:
                attempt += 1
                logger.error(f"API error on batch {current_batch}: {e}. "
                             f"Retry {attempt}/{max_retries}")
                if attempt >= max_retries:
                    raise
                time.sleep(retry_delay)

    logger.info(f"Embedding complete. Total vectors: {len(all_embeddings)}")
    return all_embeddings

Running Inference Locally with sentence-transformers

Some organisations face data residency constraints that rule out sending policy documents to an external API. In those cases, local inference with sentence-transformers is the go-to solution.

from sentence_transformers import SentenceTransformer
import numpy as np
from typing import List, Optional
import logging

logger = logging.getLogger(__name__)

class LocalEmbeddingPipeline:
    """
    Production-ready local embedding pipeline using sentence-transformers.
    Suitable for data-residency-constrained banking environments.
    """

    def __init__(
        self,
        model_name_or_path: str,
        device: str = "cpu",
        batch_size: int = 64,
        normalise: bool = True
    ):
        self.model = SentenceTransformer(model_name_or_path, device=device)
        self.batch_size = batch_size
        self.normalise = normalise
        self.device = device
        logger.info(f"Loaded model: {model_name_or_path} on {device}")

    def embed(
        self,
        texts: List[str],
        show_progress: bool = True
    ) -> np.ndarray:
        """
        Embed a list of texts. Returns an (N, D) numpy array.
        Normalises to unit length if normalise=True (required for cosine similarity).
        """
        embeddings = self.model.encode(
            texts,
            batch_size=self.batch_size,
            show_progress_bar=show_progress,
            normalize_embeddings=self.normalise,
            convert_to_numpy=True
        )
        logger.info(f"Embedded {len(texts)} texts. "
                    f"Output shape: {embeddings.shape}")
        return embeddings

    def embed_query(self, query: str) -> np.ndarray:
        """Embed a single query. Returns a 1D array."""
        return self.embed([query], show_progress=False)[0]

Checking Token Length Before You Embed

If a chunk runs longer than a model's token limit, it gets truncated without warning. In a policy corpus, that silent truncation can strip out exactly the clause or numeric threshold that made the chunk worth retrieving in the first place. Validating token counts before embedding catches this problem before it ever reaches your index.

from transformers import AutoTokenizer
from typing import List, Tuple
import logging

logger = logging.getLogger(__name__)

def validate_chunk_lengths(
    chunks: List[str],
    model_name: str,
    max_tokens: int,
    truncation_strategy: str = "warn"
) -> Tuple[List[str], List[int]]:
    """
    Validate that all chunks are within the model's token limit.

    truncation_strategy:
        "warn"  - Log a warning for oversized chunks and include them (will be truncated by model).
        "skip"  - Remove oversized chunks and return only valid ones.
        "raise" - Raise ValueError on the first oversized chunk.

    Returns (validated_chunks, oversized_indices).
    """
    tokeniser = AutoTokenizer.from_pretrained(model_name)
    oversized = []

    for idx, chunk in enumerate(chunks):
        token_count = len(tokeniser.encode(chunk, add_special_tokens=True))
        if token_count > max_tokens:
            oversized.append(idx)
            msg = (f"Chunk {idx} has {token_count} tokens, "
                   f"exceeds model limit of {max_tokens}. "
                   f"First 80 chars: {chunk[:80]!r}")
            if truncation_strategy == "raise":
                raise ValueError(msg)
            else:
                logger.warning(msg)

    if truncation_strategy == "skip" and oversized:
        valid = [c for i, c in enumerate(chunks) if i not in set(oversized)]
        logger.info(f"Removed {len(oversized)} oversized chunks. "
                    f"{len(valid)} chunks remain.")
        return valid, oversized

    return chunks, oversized

Attaching Metadata and Provenance to Embeddings

A raw embedding vector isn't enough on its own for a regulated RAG system to function correctly. Each vector needs structured metadata attached to it so that retrieval, reranking, and generation stages downstream can confirm where the content came from, enforce access permissions, restrict results by jurisdiction, and point back to an authoritative source document.

Going back to the EUR 12 million credit proposal scenario, every embedded chunk should include, at a minimum, the fields shown here:

from dataclasses import dataclass, field
from typing import Optional, List
import uuid

@dataclass
class EmbeddedChunk:
    """
    Production embedding record for a banking policy RAG system.
    The vector enables retrieval. The metadata enables everything else.
    """
    # Vector
    vector: List[float]
    vector_dimensions: int
    embedding_model: str
    embedding_model_version: str

    # Content
    text: str
    content_type: str          # "narrative", "table_row", "proposition", "image_description"

    # Provenance
    document_id: str
    document_version: str      # e.g. "7.2"
    policy_id: Optional[str]   # e.g. "CRD-EU-047"
    jurisdiction: Optional[str] # e.g. "EU"
    effective_date: Optional[str]

    # Chunk structure
    chunk_id: str = field(default_factory=lambda: str(uuid.uuid4()))
    parent_id: Optional[str] = None
    section: Optional[str] = None
    page_number: Optional[int] = None
    source_artifact_path: Optional[str] = None  # path to original image/table

    # Access control
    classification: str = "INTERNAL"            # "PUBLIC", "INTERNAL", "CONFIDENTIAL"
    permitted_roles: List[str] = field(default_factory=list)

    # Indexing
    indexed_at: Optional[str] = None
    indexing_pipeline_version: Optional[str] = None

Carrying this metadata consistently through every stage of the pipeline, from chunking to embedding to vector indexing, isn't just good practice. In a regulated banking context, retrieving a technically correct answer that's grounded in an outdated policy version counts as a compliance failure. The vector's job is to find the chunk; the metadata's job is to confirm that the chunk came from the correct, current version of the source.

Evaluating Embedding Quality

Public leaderboards such as MTEB report general-purpose retrieval scores across a wide range of academic datasets. Those numbers are helpful for weeding out models that are clearly weak performers. They fall short, however, when the task is picking the best model for a specialised corpus like a bank's internal policy library.

The only measurement that truly matters is how a model performs against your own documents, using your own queries, scored against relevance judgments you've labelled yourself.

Building a Retrieval Evaluation Set

A retrieval test set built for a banking RAG embedding pipeline needs to cover several query types:

Narrow factual lookups that map to one authoritative chunk, such as asking how often high-risk corporate customers must undergo their annual review at minimum.

Threshold-based questions that combine a specific numeric condition with the governance rule attached to it, such as asking which approval authority is required once an EU corporate facility exceeds EUR 10 million.

Multi-evidence questions where a full answer depends on pulling together more than one chunk, such as asking what checks must happen before a high-risk corporate credit proposal can even be submitted.

Synthesis questions that draw content from several sections at once, such as asking for a full description of the AML control framework governing high-risk corporate lending.

Cross-document questions, relevant wherever policies reference one another across separate documents.

import numpy as np
from typing import List, Dict, Set

def recall_at_k(
    retrieved_ids: List[str],
    relevant_ids: Set[str],
    k: int
) -> float:
    """
    Compute Recall@k for a single query.
    relevant_ids is the ground truth set of chunk identifiers.
    retrieved_ids is the ordered list of retrieved chunk identifiers.
    """
    if not relevant_ids:
        return 0.0
    top_k_retrieved = set(retrieved_ids[:k])
    return len(top_k_retrieved & relevant_ids) / len(relevant_ids)

def mean_reciprocal_rank(
    retrieved_ids: List[str],
    relevant_ids: Set[str]
) -> float:
    """Compute MRR for a single query."""
    for rank, chunk_id in enumerate(retrieved_ids, start=1):
        if chunk_id in relevant_ids:
            return 1.0 / rank
    return 0.0

def evaluate_embedding_model(
    model_name: str,
    evaluation_queries: List[Dict],
    corpus_chunks: List[Dict],
    k_values: List[int] = [1, 5, 10, 20]
) -> Dict:
    """
    Evaluate an embedding model on a labelled retrieval dataset.

    evaluation_queries: List of dicts with "query" and "relevant_chunk_ids" keys.
    corpus_chunks: List of dicts with "chunk_id" and "text" keys.
    Returns per-query-type and aggregate retrieval metrics.
    """
    from sentence_transformers import SentenceTransformer

    model = SentenceTransformer(model_name)

    corpus_texts = [c["text"] for c in corpus_chunks]
    corpus_ids = [c["chunk_id"] for c in corpus_chunks]
    corpus_embeddings = model.encode(corpus_texts, normalize_embeddings=True)

    results_by_type: Dict[str, List] = {}
    all_recall: Dict[int, List[float]] = {k: [] for k in k_values}
    all_mrr: List[float] = []

    for query_item in evaluation_queries:
        query = query_item["query"]
        relevant = set(query_item["relevant_chunk_ids"])
        query_type = query_item.get("query_type", "unspecified")

        query_embedding = model.encode(query, normalize_embeddings=True)
        scores = corpus_embeddings @ query_embedding
        ranked_indices = np.argsort(scores)[::-1]
        retrieved = [corpus_ids[i] for i in ranked_indices]

        mrr = mean_reciprocal_rank(retrieved, relevant)
        all_mrr.append(mrr)

        for k in k_values:
            r = recall_at_k(retrieved, relevant, k)
            all_recall[k].append(r)

        if query_type not in results_by_type:
            results_by_type[query_type] = {"mrr": [], "recall": {k: [] for k in k_values}}
        results_by_type[query_type]["mrr"].append(mrr)
        for k in k_values:
            results_by_type[query_type]["recall"][k].append(
                recall_at_k(retrieved, relevant, k)
            )

    aggregate = {
        "model": model_name,
        "n_queries": len(evaluation_queries),
        "mrr": float(np.mean(all_mrr)),
        "recall": {k: float(np.mean(all_recall[k])) for k in k_values}
    }

    per_type = {
        qt: {
            "mrr": float(np.mean(data["mrr"])),
            "recall": {k: float(np.mean(data["recall"][k])) for k in k_values},
            "n_queries": len(data["mrr"])
        }
        for qt, data in results_by_type.items()
    }

    return {"aggregate": aggregate, "by_query_type": per_type}

Retrieval metrics shouldn't be judged in a vacuum, separate from how they affect the quality of the final answer. Suppose Recall@5 improves by three points but the rate of near-duplicate chunks being retrieved climbs by 30 percent — that trade-off may not actually help the LLM, since feeding it three nearly identical passages instead of one useful passage doesn't add real evidence.

The right approach is to assess the whole chain, from the initial query through to the final answer that's presented to the user. An embedding model's role is limited to getting the right evidence into the context window. Whether that evidence then leads to an answer that's both accurate and compliant is determined by every stage that comes after it.

The Embedding Pipeline as Infrastructure

Once a chunk finishes moving through the embedding pipeline, it needs to come out the other side with a vector attached, metadata fully populated, and a stable identifier that lets you retrieve, update, or delete it later without corrupting the records sitting next to it in the index.

This is a piece of infrastructure, not a one-off script. Production-grade embedding pipelines for regulated banking environments require the following:

• Idempotency. If a chunk gets re-embedded because you switched model versions, that should overwrite the existing record, not spawn a duplicate.
• Version tracking. Whenever the embedding model changes, you either rebuild the index from scratch or partition it cleanly by model version. Letting vectors from different models coexist in the same index gives you similarity scores you can't trust.
• Propagation of access controls. If a chunk was tagged CONFIDENTIAL during ingestion, that tag has to survive embedding and land in the vector index intact. The retrieval layer then needs to honor it.
• Observability. You should be logging embedding latency per batch, token counts, API error rates, and chunk-level failures, all with structured metadata attached. A truncation that silently alters retrieval behavior is exactly the kind of thing your monitoring should catch, not something invisible.
• Cost tracking. Embedding costs through an API add up fast once you're operating at scale. Break down spend by document type and by pipeline run, so you can actually weigh model choice and dimensionality reduction against real costs rather than guesswork.

None of this is optional once you're in production. These are the qualities that separate a pipeline that merely works in a demo from one you can operate, audit, and maintain over time inside a regulated environment.

Return to the EUR 12 Million Credit Proposal

The relationship manager's original question hasn't changed: what approval authority does this deal need, and which controls have to be cleared before it can be submitted?

• Part 4 covered how to design retrieval units that keep those answers intact in their original form: the EDD policy clause, the approval-matrix row specifying GCC authority, the workflow diagram laying out pre-submission controls.
• In this part, we've converted those retrieval units into searchable vectors, using a model that was benchmarked against banking-specific terminology, fine-tuned to grasp how regulatory abbreviations relate to their compliance context, and embedded alongside full provenance metadata that lets retrieval confirm policy version and jurisdiction later on.

When the query comes in, the vector index locates the approval-matrix row covering high-risk exposures above EUR 10 million, the EDD clause, and the diagram showing the control sequence — and it hands them back along with metadata confirming that all three trace to policy CRD-EU-047, version 7.2, EU jurisdiction, effective January 15, 2026.

What reaches the generation layer is evidence that's accurate, complete, and traceable.

That's the distinction between an embedding pipeline that simply loads chunks into a vector database and one that preserves everything needed for answers that are reliable and auditable.

Before You Move to Vector Indexing: A Checklist

Before letting embedded chunks into the vector index, confirm the following:

• Is the input type parameter set correctly on the embedding model for both indexing calls and query calls? Getting this mismatched quietly erodes retrieval precision.
• Has every chunk's token length been checked before embedding? Silent truncation changes what the embedded text actually represents, with no error raised anywhere in the pipeline.
• Does each vector record include full provenance — document version, policy ID, jurisdiction, and effective date?
• Was the embedding model actually tested against your own domain corpus and query patterns, rather than chosen purely on the strength of public benchmark rankings?
• If you fine-tuned the model, do you have before-and-after retrieval numbers measured on a held-out query set?
• Are your quantisation decisions backed by results from your own retrieval evaluation set, rather than assumptions carried over from published benchmarks?
• Is the pipeline idempotent, version-aware, and observable to the standard a regulated production system demands?

Miss any of these and the vector index won't catch it for you — it will happily store whatever you feed it. Nothing in the database will flag a vector built from a truncated chunk, a query embedded with the wrong input type, or a chunk that was embedded using a model version out of sync with the rest of the index. Those gaps don't announce themselves; they resurface later as retrieval quality issues that look, from the outside, like problems with the LLM.

Earlier in this series, Part 4 showed how to turn clean documents into retrieval units while keeping their structure intact. This part has shown how to turn those retrieval units into searchable vectors while keeping their provenance intact.

What comes next is where those vectors actually meet the index: dense vector search, sparse retrieval, hybrid approaches, approximate nearest-neighbor algorithms, and the metadata filtering that decides which vectors are even in the running for retrieval in the first place.