grep Beat Vector Search in Agent Memory Tests, and the Harness Mattered as Much

Retrieval-augmented systems have long rested on one belief: vector retrieval beats keyword search because it captures meaning. A request for "documents about quarterly earnings" finds a "Q3 revenue report" even though the two phrases share no words, since the embedding space treats them as close. That belief has driven an enormous amount of infrastructure spending, from Pinecone and Weaviate to Chroma and Qdrant. Early work on dense passage retrieval showed it beating sparse BM25 on open-domain question answering, the field took the result on board, and vector search became the default.

A 2026 study set up a different experiment. Agents were given a retrieval task from LongMemEval, a benchmark in which an agent has to answer questions by searching through long conversation histories. The test pitted grep, plain exact string matching, against vector retrieval. It ran across several agent harnesses: a custom one called Chronos, plus the provider-native command-line tools Claude Code, Codex and Gemini CLI.

In the main setup grep came out ahead of vector retrieval on accuracy, and it did so on every harness, not just one.

The short version

• Setup: 116 LongMemEval questions, grep versus vector retrieval, four harnesses, identical conversation data.
• Main result: with results returned inline, grep beat vector retrieval on accuracy everywhere it was tried.
• Second finding: the harness mattered about as much as the retrieval method. The same retrieval approach produced very different accuracy on different harnesses.
• Why grep wins here: the questions ask for specific facts: names, dates, exact phrases. Semantic similarity adds noise to that kind of lookup, because grep returns the string itself while vector search returns passages that merely resemble it.
• The harness issue: how a tool result is shown to the model decides whether the model uses it properly. A perfect result presented badly is wasted.
• The awkward question: how much of the advantage credited to better retrieval in production RAG is really explained by better harness design?

Why grep suits memory questions

The mechanism is easy to state. LongMemEval asks things like what someone named as their favorite restaurant in a conversation on a given date, or the title of a book a person recommended. Each answer is a particular string somewhere in the history.

Vector retrieval embeds the question, finds conversation segments with nearby embeddings and returns them. The trouble is that being close in meaning isn't the same as holding the exact fact. A segment about food in general sits near "favorite restaurant" in embedding space. It may never name the restaurant, yet it ranks high and contains no answer.

Searching with grep for "restaurant" returns every segment that includes the word, and that includes the one where the favorite was mentioned. grep is exact, vector retrieval is approximate, and for exact-fact lookups approximation just adds noise.

Vector retrieval:  high recall on semantically related segments
                   lower precision on exact-fact queries
                   noise from semantically similar but fact-absent segments

grep:              perfect precision on string matches
                   lower recall on paraphrased or implicit facts
                   no noise from semantic neighbors

The benchmark's questions are worded to demand exact recall, so the evaluation distribution favors exact string matching. This isn't proof that grep is better than vector retrieval in general. It shows that the task distribution decides which method wins. The question worth asking is what the real mix of retrieval tasks looks like in production agent deployments, and which method wins on that mix.

The harness effect

The bigger finding isn't grep against vectors. It's the harness. With the same retrieval method, the same data and the same questions, accuracy shifts a lot depending on the harness. The reported numbers show the scale: one Claude Opus model reaches 93.1% under Chronos and 76.7% under Claude Code, so swapping harness moves the ceiling by roughly as much as swapping the retriever inside a fixed harness. The widest inline gap between grep and vector was on Chronos with Gemini 3.1 Flash-Lite, at 86.2% against 62.9%.

A harness does four things:

1. It calls the retrieval tool with a query.
2. It receives the results.
3. It presents those results to the model.
4. The model reasons over them to answer.

For a given retrieval method, steps one, two and four are the same everywhere. Step three differs, and the way results are presented changes what the model does with them. Presentation choices include these:

• Format: a ranked list, one best hit or everything concatenated.
• Context: whether each hit carries its source, timestamp and surrounding text.
• Highlighting: whether the relevant portion is marked or the model must hunt inside it.
• Filtering: whether low-confidence hits are dropped or all are shown.

Nobody has standardized these choices, and each provider's CLI makes its own. The delivery style turned out to matter a great deal. Handing results back inline favored grep, while passing them as files the model reads separately reshuffled the comparison: vector retrieval overtook grep on about half the harness and model pairings. One Codex setup fell from 93.1% with inline grep to 55.2% when grep results were delivered as files.

If so, studies that compare retrieval methods while holding the harness constant may be measuring the wrong variable. A better retriever inside a worse harness can lose to a worse retriever inside a better one.

Mixing in distractors

A second experiment started from the correct conversation passages and mixed in growing amounts of unrelated conversation history, then tracked accuracy as the clutter increased.

Both methods lose accuracy as distractors grow, which is expected. The curves differ, though. On exact-fact queries grep degrades more gracefully. When the relevant passage contains the query string, grep finds it however many distractors surround it, since the distractors don't match the string and never show up in the results.

Vector retrieval declines faster. Distractors that resemble the query semantically do appear in the results, so the share of useful hits in the top results shrinks as the clutter grows, and the model has to dig the answer out of a noisier set.

Accuracy = f(retrieval_precision × harness_quality)

grep retrieval precision: high for exact facts, stable with distractors
vector retrieval precision: moderate, degrades with semantically similar distractors

For agents working on large, mixed context stores, with long conversation histories, several document types and uneven relevance, that resistance to distractors is a real practical advantage on exact-fact tasks.

What this means for the retrieval literature

Papers that compare dense with sparse retrieval, embedding models or reranking strategies usually fix the harness and vary the retriever, treating the harness as a constant. The findings here say it isn't a constant. It interacts with the retrieval method. A harness tuned for vector retrieval may present grep results poorly, and the reverse is true too.

It's possible that retrieval methods have been tuned against harnesses that happen to favor vector approaches. The absolute comparisons could still hold while the relative rankings don't.

In practice, teams building agentic RAG spend a lot of effort picking embedding models, vector database vendors and similarity metrics. Few spend comparable effort on the harness that shows retrieval results to the model. These results suggest the harness deserves at least as much engineering attention as the retriever, but engineering practice hasn't caught up.

Reading the results with care

This line of work probably won't get the attention it merits, and not because the results are weak. They're inconvenient for an industry built on vector search, since grep sometimes wins and harness design is undervalued.

They should still be read carefully. LongMemEval has a specific task distribution that favors exact-fact retrieval, and grep's lead there doesn't mean it wins everywhere. Semantic retrieval does better on paraphrase-heavy tasks, cross-lingual retrieval and concept-level queries. The study also looked at verbatim multi-session chat evidence, and hybrid retrieval or stronger lexical baselines weren't part of it.

The takeaway isn't to use grep. Match the retrieval method to the distribution of retrieval tasks, and engineer the harness as carefully as the retriever. Neither is standard practice yet, and both should be.

Questions for your own system

For the agentic retrieval tasks in your production system, what fraction need exact-fact recall and what fraction need semantic concept matching? And when did you last review how your harness presents results with the same care you gave your choice of embedding model?