Prompt Caching Makes Self-Consistency Cheap for Long-Context LLM Checks

Why consistency checks don't have to cost six times as much

A cheap, portable way to catch an unreliable LLM has been around for years. Put one question to the model repeatedly and compare what comes back. If the answers disagree, don't trust any of them. It needs no logprobs, labels or judge model, and any API you can call twice supports it. In production, though, it's rare, because everyone raises the same objection: six samples mean six times the bill.

Testing on six models shows a different picture. Where prompt caching is active, six samples land at 1.55 to 2.5 times the price of one call. A clarification on what that means: the numbers are ratios, so a scored request still costs more than a plain one. What changes is the size of the surcharge, around 55% instead of the 500% people picture. Absolute dollar figures are in the tables so you can verify this.

A workload caching was built for

The reason is mechanical. A request reaches the API as a single stream of tokens, context first and question last. Caching compares that stream with earlier ones starting at byte zero, and whatever matches is billed around 90% below the normal input rate. A change anywhere early in the prompt knocks out everything behind it.

Consistency sampling never changes anything. It repeats the exact same prompt N times. The first call pays the full rate and fills the cache, and the remaining five reuse the context at a tenth of the rate, so the only new work is generating answers, the very thing you want to compare. For RAG over long documents or agents carrying large tool contexts, which are the cases where trust matters most, input tokens make up most of the cost, and so the 6x largely vanishes.

Some details complicate this. Writing to Anthropic's cache costs a one-time 1.25x surcharge, and Anthropic is the only one of the three providers that reports that write. Gemini caches in blocks of 8,192 tokens, and in these runs nothing was cached between about 9k and 17k tokens. Minimum sizes also vary, at 512 tokens for Claude Opus-5, 1024 for OpenAI and 4,096 for Gemini.

The test setup and what it cost

The test used a roughly 21k-token airline policy as context and one question after it, sent six times to each model. After each call, the provider's cached-token count was logged, and four providers use four different field names for it. UQLM's noncontradiction scorer then rated the six answers, using an NLI model to compare each answer with the other five. That step runs locally after the fact and adds no API calls.

The folk assumption is 6x. With the cache warm, cost for a scored request divided by an unscored one came out between 1.55x and 2.5x for most models, measured on 31 July 2026.

Two results undercut the optimism. Qwen reported no cache hits on any of its six calls, which puts it at exactly 6x, so for that model the objection holds as stated. Llama cached 99.4% of the prompt yet saved nothing, because Together lists a discounted cached rate for some models but not this one. Hitting the cache is a technical event and getting a discount is a business decision, so you need both to happen.

Longer context, smaller surcharge

To test the "long-context" claim, the sweep ran from 1.4k up to 440k tokens, three calls per size, each size on its own cold prefix so nothing inherited a warm cache from the previous run.

Every curve drops and then levels off at a floor. That's a genuine asymptote and not a slow decline. OpenAI measures 1.51x, 1.51x, 1.50x and 1.50x from 55k through 440k tokens, so an eightfold increase in context moves the result by a hundredth. To go lower than the floor you'd need fewer samples (three samples bottom out at 1.20x) or a larger cache discount.

Two providers deviate in their own ways. Anthropic bottoms out at 1.75x and not 1.50x, since each cold call pays the 1.25x cache write. Gemini's 8,192-token blocks make its multiplier wobble whenever the prefix crosses a block boundary: 2.07x at 28k, up to 2.31x at 50k, then 1.63x at 84k, all heading toward the same floor.

Under roughly 7k tokens, Gemini cached nothing and stayed at a flat 6.00x. That defines where the argument stops working: with small prompts, the standard objection is simply correct.

One equation covers it

Everything reduces to one line. For N samples of the same prompt:

C  cached context tokens
c  cached / input rate   (~0.1; 1.0 if no cached price)
Q  uncached input per call
w  write / input rate    (1.0; 1.25 on Anthropic)
O  output tokens
k  output / input price  (6 Terra · 5 Opus, Gemini · 2 DeepSeek)

In this formula, f measures how much of the request cost can ride the cache, and M is linear in it, equal to N when nothing is cached and equal to the floor when everything is. All six models fall on that line within 0.005.

What deserves attention is the cost weighting. Output tokens count in f after multiplying by k, because they cost k times more and are never cached. For Terra, a single output token erodes the cacheable share as quickly as six context tokens add to it, which explains most of the observed spread. DeepSeek reaches 1.86x even with a 99% hit rate, because each answer included 656 tokens of reasoning. Llama sits at exactly 6.00x with f = 0.99: with no cached price, c = 1, and the line gives it no room.

The practical rule is that cached context C should far exceed k times the output O. Long context helps, and long answers hurt k times as fast. A surface plotted over context and output length puts the 2x break-even where those two forces balance.

For your own stack this is arithmetic and not a benchmark:

# pip install r2c2 — Reuse Context, Recheck Consistency
from r2c2 import estimate, required_context
estimate(context_tokens=21_000, output_tokens=40, model="gpt-5.6-terra").multiplier
# 1.56 — matches the 1.55 measured
required_context(output_tokens=40, model="gpt-5.6-terra", threshold=1.8)
# 4060 — under ~4k context tokens, six samples cost more than 1.8x

The second function runs the formula in reverse. Give it a cost ceiling and it returns the context size at which scoring begins to meet that ceiling. It returns None if no context size can, which happens when the ceiling is at or under that provider's floor.

What the extra cost catches

Cheap sampling is pointless unless the score actually flags something, so the test question was a hard one. The policy allows refunds on non-refundable fares within 24 hours of booking, and it doesn't say who cancelled. The scenario: a passenger booked a non-refundable fare 30 hours earlier, and then the airline cancelled the flight. Is a refund due?

On a control question the policy answers directly, all six models scored 0.99 or higher. The wording changes each time and the verdict never does, because rewording is what sampling is and doesn't count as disagreement.

On that question, drawn from the same six calls used for the cost figures, the models disagreed. Qwen alternated No, Yes, No, Yes, No, Yes, which is a coin toss on a refund decision, and it scored 0.40. Gemini split 4 to 2, and its yes answers cited "Department of Transportation regulations" that the prompt never mentions. Terra and DeepSeek each split 5 to 1. Claude Opus 5 answered yes six times out of six, having decided, plausibly yet without textual support, that the 24-hour rule covers only passenger-initiated cancellations. Llama earned a perfect 1.00 by giving the same hedge six times ("may be considered an exception... may be eligible"). Hedging consistently is a signal too, but this scorer doesn't flag it.

Every one of the thirty-six answers reads as fluent and confident, and each cites the policy. Looking at one response alone, you'd never learn that the next sample might say the opposite. The 0.40 is the sole visible warning, and where input dominates, getting it costs roughly half of one additional call.

The splits also vary from run to run. In another run Terra split 3 to 3 and Gemini declined all six times. Testing beforehand can't fix that, which is the case for scoring live and not auditing once offline.

Putting it to work

If you already sample N responses, the scoring itself is free, since .score() runs after the fact on answers you hold:

from uqlm import BlackBoxUQ
# six answers you already collected; calls 2-6 hit the prompt cache
scorer = BlackBoxUQ(scorers=["noncontradiction"], use_best=False)
result = scorer.score(responses=[answers[0]], sampled_responses=[answers[1:]])
confidence = result.to_df()["noncontradiction"][0]  # 0.99 = agrees; 0.40 = coin flip

The scorer runs a local NLI model (microsoft/deberta-large-mnli), so there's no judge-model fee. Low scores can go to a person, to another retrieval attempt or to a larger model.

Three habits determine whether you really see 1.55x. Send the first call by itself and fan out the rest, because a cache entry can't be read until the first response begins streaming, and six simultaneous identical requests would all miss. Keep the samples identical down to the byte and put anything volatile after the context, since a single interpolated timestamp spoils everything that follows. Finally, sample with a temperature above zero, otherwise six identical answers give you a perfect score that tells you nothing.

Where this leads

Among hallucination-detection methods, black-box uncertainty estimation was always the easiest to deploy, needing no logprobs or labels and working anywhere. Its one serious drawback was price, and that drawback faded once providers introduced cached reads at a 90% discount, since consistency sampling resends identical bytes by design. The loop is: cache the reading, resample the answer, score agreement, and send low scores to a person. On the platforms carrying most production traffic, it costs about one-half of an extra call.

None of this carries over unchanged to every setup. The figures depend on context size, provider and token prices. A 2k-token prompt isn't cached at all on Gemini, and a chatbot with short prompts and long answers won't reach 1.55x. The equation gives your own multiplier in half a minute, and the reproduction repository redoes all of it for four API keys and around two dollars.

Still, the direction suits this technique, and the reason is agents. Their contexts are the fastest-growing prompts in production, because each tool call adds to the history and each turn resends all of it, and even careful compaction doesn't stop the totals from climbing. Growth doesn't weaken the argument, it moves you further along the curve. Providers built prompt caching since agents resend huge prefixes, and consistency scoring uses that same machinery. Verification is getting cheapest in the agent workloads where it's most needed.

Code and figures: https://github.com/mohitcek/r2c2 (the reproduction repo) and https://github.com/cvs-health/uqlm (UQLM).