Choosing an Embedding Model for RAG: Why Switching Cost Beats Price

In 2026, embedding model comparisons are often done carefully yet settle a question that barely matters. Spreadsheets list price per million tokens. Retrieval tests run side by side on two hundred sample documents. A week goes on vector dimensions. Then a model is chosen, the system ships, and half a year on the team meets a cost nobody budgeted for.

Here's an argument that sounds wrong at first but holds up. For most production RAG systems, the monthly spend gap between hosted embedding options is close to a rounding error. The figure that actually shapes the architecture is what it costs to change course later. The arithmetic is the whole case, so it comes first.

Three axes, usually weighted backwards

Embedding models differ in three ways, and teams often rank them in the wrong order.

• Retrieval quality gets benchmarked first. It's measured publicly on MTEB, and it's real, but the spread among the leaders is much tighter than marketing implies.
• Cost goes into the slide deck. It's normally quoted per million tokens, which makes it look like the dominant variable. It isn't, as the numbers below show.
• Operational surface almost never gets scored. Who runs the model? What happens when the provider retires it? How long does a full index rebuild take? Can the model run at all once compliance says documents must stay inside the network?

The claim: quality gaps at the top are too small to change a product outcome, cost gaps are too small to change a budget, and operational surface is the axis that quietly decides how locked-in you end up.

What embedding actually costs

Pricing anchors everything, so start there. OpenAI's text-embedding-3-small is priced at $0.02 per million tokens, making it one of the most cost-effective hosted options in 2026.

Now apply that to a realistic workload. Take a corpus of a hundred million tokens, which is a seriously large internal knowledge base, and index all of it:

corpus_tokens      = 100_000_000
price_per_million  = 0.02            # text-embedding-3-small
index_build_cost   = corpus_tokens / 1_000_000 * price_per_million
# 2.0

Two dollars. That's the entire API bill for embedding a hundred-million-token corpus once.

The query side looks similar. At five hundred thousand queries a month on text-embedding-3-small, embedding the queries costs about $3.25 monthly. Next to an inference bill, a vector store bill and a single engineer's afternoon, that's noise.

What about the bigger text-embedding-3-large? It costs more per token and scores higher on public benchmarks. The honest rule for reaching for it: only after measuring a retrieval failure that better embeddings would fix. If the top result is wrong because chunks are badly cut, or because queries don't resemble documents in cosine-similarity terms, a stronger encoder won't help. It will only make a wrong retrieval a bit more confident.

The same reasoning applies to the older ada-002 versus 3-small question. 3-small is cheaper and generally stronger, so a greenfield project has an easy answer. With an index already built, price doesn't enter into it, as the migration section below explains.

OpenAI versus Cohere

This comparison comes up in nearly every architecture review, and it's usually framed as quality when it's really a question of fit. One caution: current published Cohere pricing and benchmark figures aren't available here, so none are invented. What can be said is where the shape of the decision differs.

Cohere positioned Embed v3 around multilingual retrieval and search relevance, not around being the cheapest general-purpose encoder. For a truly multilingual corpus, especially one where users query in one language against documents in another, that's the axis to test on your own data instead of trusting a leaderboard. Cross-lingual retrieval is the place where model choice still changes what users see, because a model that handles it poorly doesn't degrade gracefully. It returns documents that are confidently irrelevant.

The practical instruction is narrow. A monolingual English workload makes the comparison mostly academic, so decide on operational grounds. Anyone else should build an evaluation set of around a hundred queries in the real languages and run it before committing, since this is where benchmarks generalise worst.

Local models change the question

One figure changes the picture: local embedding models such as Nomic Embed and BGE come within five percent of text-embedding-3-large on MTEB, and they cut per-token cost to zero at high volume.

That's five percent on a public benchmark, one that already exaggerates how far the choice of model moves results on any given corpus.

Running a model locally therefore isn't about saving money. The hosted API bill was already tiny, and self-hosting a sentence-transformers-style model on your own GPU has costs too: the instance, the deployment, and whoever is on call for it at three in the morning.

Self-hosting wins on three counts, all operational:

1. Data residency. If documents can't leave your infrastructure, the debate is over, and the only open question is which local model.
2. Predictable throughput. Nothing limits requests except settings you chose, and there's no metered bill to forecast. A pipeline that rebuilds its index around the clock gains more from that stability than from skipping the token fees.
3. Control over your own vector space, which matters most and gets discussed least.

The real switching cost

Each vector in an index came from one specific model. Vectors from different models can't be compared, and there's no conversion between them. Change the model and every stored vector turns meaningless against new queries.

So a model swap means rebuilding the whole index. The usual cost framing misleads people here. Scary sticker prices get quoted for re-embedding a large corpus, yet against real token pricing the API portion is small. Take a hundred million tokens at --.02 per million: that's two dollars, and a model priced ten times higher still leaves the total in the low hundreds.

The true cost isn't a line in a token budget. It's everything around it:

• The pipeline that must pull every document out of storage again and apply the same chunking scheme, which may not have been preserved.
• The parallel index that has to run while both versions are live, since a production vector store can't be swapped atomically during business hours.
• The retrieval evaluation, which has to be redone from scratch because every relevance threshold was tuned against the old vector space.
• Two or three weeks of engineering attention, landing in whichever quarter the problem surfaces.

That's the trap. It doesn't arrive as an unexpected invoice. It arrives as an unexpected project.

That suggests a simple rule for choosing: prefer the model that is least likely to be pulled out from under you. Provider track record, a clear deprecation policy and the option to pin a version count for more than one extra benchmark point, because they determine whether a migration is ever forced. Keeping a marginally weaker model for four years is better than adopting a marginally stronger one and dropping it after eighteen months.

A quick decision guide

• English corpus, new build: text-embedding-3-small. It's the cheapest credible start, and its cost is negligible at real volume.
• Measured retrieval failures that a stronger encoder would fix: text-embedding-3-large, but only once there's evidence.
• Multilingual or cross-lingual retrieval: Cohere Embed v3, evaluated on your own data, since model choice still shows up in the user experience here.
• Documents that can't leave the network: self-hosted Nomic Embed or BGE. Compliance decides this one, not cost.
• Very high continuous re-indexing volume: a local model, which takes rate limits and per-request billing out of the design.

Only two of those rows are settled by the model itself. Constraints that exist whichever model you choose decide the rest.

Common questions

What's the cheapest embedding model in 2026? Among hosted choices, text-embedding-3-small at --.02 per million tokens ranks near the bottom on price while staying credible. Local options like Nomic Embed and BGE have no per-token fee at all, though you pay in hardware and ownership instead. Either way the amounts are small enough that cost shouldn't settle the matter for most teams.

Should embeddings come from OpenAI or Cohere? For English-only workloads, decide on operational grounds, since the quality gap won't be what limits the product. For multilingual or cross-lingual workloads, build an evaluation set in the actual languages and test both, because that's where users see a difference.

How much does re-embedding a large corpus cost? Less in API fees than most people guess: about two dollars for a hundred million tokens at --.02 per million. What hurts is the labour around it, which includes re-extracting and re-chunking each document, keeping two indexes alive through the cutover, and repeating every retrieval evaluation in the new vector space. Plan in weeks rather than tokens.

The takeaway: do the model comparison, but spend the anxiety saved on the migration path rather than the price sheet. Choosing a model is a smaller decision than it feels like. The index built around it is a far bigger one.