Gemini's 1M Window Replaced My Vector Store; Latency and Provenance Objected

Dropping the vector table

Picture a support assistant used by nine thousand internal staff. Every answer it produces is something a human agent might pass on to a customer, so wrong or unexplainable output is expensive.

The retrieval layer underneath is made of a chunker, an embedding job, a vector store and, last in line, a reranker that has been adjusted so many times nobody can justify its numbers anymore.

user ─► embed ─► vector db ─► rerank ─► 4k ctx ─► model ─► answer
                    ▲
                 chunker
                    ▲
              nightly reindex

Running it means four services and two on-call alerts, plus a recurring bug where a PDF footer quietly corrupts half the chunks. The query path itself is tiny:

q = embed(text)
hits = db.search(q, k=8)
ctx = "\n".join(h.doc for h in hits)
ans = llm(f"{ctx}\n\nQ: {text}")

Everything surrounding those four lines is where the pain lives.

Then comes a tempting observation. Gemini accepts a million tokens with near-perfect recall, and the whole corpus is only 640,000 tokens. It fits. So someone drops a table of 2.3 million embeddings on a Tuesday night and swaps the pipeline for this:

docs = load_all()
ans = gem(f"{docs}\n\nQ: {text}")

No chunker, no reindex job, no reranker holding a constant nobody can justify. It works, and for about six hours it feels like getting away with cheating on an exam.

What the next morning showed

Latency grows roughly in line with input size. That's no secret, but you don't feel it until support starts complaining. The comparison here ran 500 real questions through each setup. Accuracy came from a second model grading every answer against a hand-written reference, and sixty of those gradings were rechecked by hand to make sure the judge wasn't being lenient.

setup               p50      p95     cost/query   accuracy
-----------------------------------------------------------
RAG, 4k ctx         1.9s     3.4s      $0.004       71%
full stuff, 640k   28.7s    41.2s      $0.81        74%
cached, 640k        9.4s    15.1s      $0.11        74%

A median of 28 seconds is a long wait for a chat box. Prompt caching brings that down and is good practice, though it has limits: a minimum prompt size, a cache lifetime you pay to keep warm, and one weakness that stung in this case. The corpus changes about forty times a day, each edit invalidates the cache, and an entire afternoon went to paying to rebuild 640,000 tokens.

The question that mattered more

Accuracy rose by three points, which almost hid the real problem. A support lead asked something simple: where did that answer come from?

Retrieval gives a concrete reply, such as chunk 4,112 on page 9, last edited in March. With the full window, the model returns a confident paragraph assembled from somewhere inside 640,000 tokens, and there's nothing to show her. Call it the provenance test. The issue isn't whether the model can answer but whether you can prove where the answer lived. That's a product problem rather than a model one, the sort that ends careers and not sprints.

Wasn't the reranker just bad?

Someone will object right away: tune the reranker, pick a cheaper tier, chunk more cleverly. Fair enough, and the reranker really was mediocre. But even a well-tuned one passes along eight passages and hopes for the best, and no amount of tuning produces a citation from a window that swallowed everything at once.

The setup that stayed

The vector database came back, and the reranker didn't.

ids = db.search(embed(q), k=40)
docs = fetch(ids)
ans = gem(f"{docs}\n\nQ: {q}", cache=c)

Retrieval no longer finds the answer. Its job became narrowing the room, and the long window does the reading.

user ─► embed ─► vector db (k=40, no rerank)
                       │
                       ▼
             ~90k ctx + cached policy docs
                       │
                       ▼
                    model ─► answer + doc ids

The same 500 questions went through the same judge:

hybrid, 90k ctx     3.1s     5.2s      $0.021       88%

Forty loose chunks give the model space to reason, where a reranker would have to guess the perfect eight. And every answer carries the IDs of what it used.

Big windows left retrieval alive and ended the precision kind. The old assignment was punishing: find the eight perfect chunks, since eight was all the budget allowed. That assignment is over, and nobody should miss it.