Reranking in RAG with Spring AI and Pinecone: Moving an Answer from Rank 4 to 1
Reranking in RAG with Spring AI and Pinecone: plain vector search vs a bge-reranker step on an HR handbook, where a buried answer moves from rank 4 to 1.
This article is published in English.
Sources: Reranking in RAG: The Spring AI and Pinecone Implementation
Type a question into a company search tool and five results come back. They all look related, yet the one that really answers the question sits third, beneath two paragraphs that merely share a few of the same words. The search understood what was said and not quite what was meant. That small gap between sounding relevant and being relevant is the reason reranking exists: vector search finds text that sounds similar, not necessarily text that answers.
What follows turns the idea into working code. Two search paths run over the same document side by side, so the difference can be watched rather than taken on faith. And it turns out to be bigger than expected.
The experiment
The test is deliberately built around a buried answer. A real HR handbook is split into small pieces, and a question is asked whose answer lives in a section nobody would guess first. The answer is then hunted for in two ways. The first looks for pieces that sound close to the question. The second takes those same pieces, judges each one properly and returns a fresh order based on what actually answers best. Seeing the two orders differ on the same question is the whole point.
Stack and prerequisites
The build runs on Java 21 with Spring Boot 3.5.14 and Spring AI 1.0.8. Embeddings come from OpenAI's text-embedding-3-small. Pinecone serves double duty as vector store and hosted rerank API, and the reranking model is bge-reranker-v2-m3.
Why use Pinecone for reranking as well? Spring AI's VectorStore has no notion of reranking, so this is the one spot that steps outside the abstraction and calls Pinecone's raw Java SDK for a single rerank call.
Three things need to exist first. One is a Pinecone account with an index whose dimension is 1536, matching the embedding model's output. Another is an OpenAI API key so embeddings can be generated. The third is Gradle to manage dependencies.
Preparing the handbook
Searching needs data first, so the handbook has to be chunked and loaded into Pinecone. One /ingest endpoint covers it: the file is read, cut into chunks of 300 tokens and stored as vectors, and a duplicate guard keeps repeated runs from inserting everything again. The file sits at src/main/resources/docs/abc_company_employee_handbook.txt. A quick skim beforehand pays off, particularly the work-from-home section, which holds one of the harder answers.
Two endpoints
The same question goes in through two endpoints that look almost identical from outside:
GET /api/search/without-reranking?query=your question here
GET /api/search/with-reranking?query=your question here
The first is the honest baseline. It queries the vector database, takes the closest matches and returns them in the order received, with no second opinion. The second does everything the first does and adds one step before returning. It passes the same chunks to a reranking model and asks which of them actually answer the question. Chunks come back reordered according to that judgment.
Both return the same response shape. Only the ordering differs, and the ordering is the point. The interesting work happens one layer below the controller, in RetrievalService.
Path 1: plain similarity search
Simplicity is the intent here: query the vector database for its nearest chunks, accept them as returned and finish. No further processing happens. One method handles the retrieval:
private List<Document> retrieveTopK(String query) {
return handbookVectorStore.similaritySearch(
SearchRequest.builder().query(query).topK(topK).build());
}
In the configuration, topK is 5, so the five nearest chunks by cosine similarity are pulled. That is the entire retrieval. Each chunk arrives with the vector store's own similarity score, showing how close it sits to the query in meaning.
Meaning is where it goes wrong. A passage can sit very close to a question and still not answer it. The sick-leave detail is tucked inside the work-from-home policy, so the surrounding WFH text looks like an excellent match, and similarity search ranks by that closeness alone. Whether the text responds to the question is never checked. What comes out is a decent shortlist with no verdict on which entry is the real answer, and the second path adds that verdict.
Path 2: adding the rerank step
The aim now is to find out which chunks actually answer the question. The path begins identically, running the same search and retrieving the same chunks:
List<Document> hits = retrieveTopK(query);
These are the identical five chunks the first endpoint used, reused on purpose. A different search here would leave no honest basis for comparing the two results.
Then comes the reranking itself. Spring AI can't help, since its vector store has no idea what reranking is, so the code calls Pinecone's hosted model directly:
return pinecone.getInferenceClient().rerank(
rerankModel, // bge-reranker-v2-m3
query,
documents, // the same five chunks
List.of("content"), // the text to judge
topN, // keep the best 3
true,
Map.of("truncate", "END"));
That is a genuine request to a hosted model, not a shortcut done locally. The reranker reads the question together with each chunk and scores how well the chunk responds. Similarity search, by contrast, compares vectors computed in advance and never reads the two texts side by side.
Picture giving the reranker five paragraphs and asking for the best three. Rather than returning text, it answers with positions, for example number 4 first, then 1, then 5, pointing at the chunks by place in line. The last small step converts those positions back into real chunks. When the model says number 4, the code returns to the original list, takes the fourth chunk and pulls out its text and score.
Document original = hits.get(ranked.getIndex());
results.add(toChunkResult(original, ranked.getScore().doubleValue()));
One subtle change: the score attached now is the rerank score, meaning how well the chunk answers the question, not the earlier similarity score. It is the same chunk, but the number beside it means something entirely different. That shift in meaning is the whole idea.
Seeing the difference
Run one question through both endpoints and read the console. The question chosen hides its answer well: what should be done after falling sick while working from home?
In the plain search, five chunks print in similarity order with their scores. The chunk that truly answers, index 6, lands at position 4 with only 37.5%. Three chunks ahead of it discuss work-from-home or sick leave in general and score higher because they share more vocabulary with the query. The real answer ranks as one of the weakest matches although it is the only complete one.
On the reranked endpoint, the same five chunks go in and the model scores each on whether it answers the question. The output adds a relevance band and a note about movement in rank. The answer chunk leaps from fourth to first at 95.6% relevance with an "Excellent match" label, while the lookalikes drop into "Weak match" at 20.4% and 6.2%. One more detail stands out. Chunk index 5, which had topped the similarity ranking, failed to reach the reranked top three. Similarity favored it, and relevance discarded it.
The takeaway
Vector search gets near the answer, and reranking lands on it. Passing the shortlist to a model that reads the question and each chunk as a pair often decides whether a user trusts what they see or scrolls on. Running the demo against private documents shows the same effect on real data, with chunks moving up and down the ranking.