Spring AI: Better RAG Results with Hybrid Search Using Lucene and RRF

Vector search is good at grasping what a query means. Lexical search is good at finding the words that were actually typed. Put the two together as hybrid search and a RAG application gets the strengths of both.

Similarity search over embeddings works well for pulling documents that are semantically close to a question, but it can be less dependable when the query contains specific terminology that matters. BM25 (Best Match 25) is a lexical ranking algorithm that does the opposite: it matches terms while weighing how often they occur, how rare they are and how long the document is. Used side by side, the two cover each other's blind spots.

The plan here is a hybrid DocumentRetriever for Spring AI that queries a vector store and a BM25 index and merges the results into a single ranked list. BM25 comes from Apache Lucene, the mature, fast search library that has been a pillar of Java search for a long time. The first change is to ingestion, so that every chunk is written to both the vector store and Lucene.

Writing chunks to the vector store and to Lucene

Start by adding Lucene to the build:

implementation 'org.apache.lucene:lucene-core:10.5.1'

Next comes a component that accepts Spring AI Document objects and indexes them in Lucene. This is LuceneDocumentWriter:

public class LuceneDocumentWriter implements AutoCloseable {

  private static final String ID_FIELD = "id";
  private static final String CONTENT_FIELD = "content";
  private static final String METADATA_FIELD = "metadata";

  private final Analyzer analyzer;
  private final Directory directory;
  private final IndexWriter writer;
  private final ObjectMapper objectMapper;

  public LuceneDocumentWriter(Path indexPath) throws IOException {
    this.analyzer = new StandardAnalyzer();
    this.directory = FSDirectory.open(indexPath);
    var config = new IndexWriterConfig(analyzer);
    this.writer = new IndexWriter(directory, config);
    this.objectMapper = new ObjectMapper();
  }

  public void add(List<Document> documents) throws IOException {
    for (Document document : documents) {
      var luceneDocument = new org.apache.lucene.document.Document();
      luceneDocument.add(
          new StringField(ID_FIELD, document.getId(), Field.Store.YES));
      luceneDocument.add(
          new TextField(CONTENT_FIELD, document.getText(), Field.Store.YES));
      luceneDocument.add(
          new StoredField(METADATA_FIELD,
              objectMapper.writeValueAsString(document.getMetadata())));

      writer.addDocument(luceneDocument);
    }

    writer.commit();
  }

  @Override
  public void close() throws IOException {
    writer.close();
    directory.close();
    analyzer.close();
  }

}

Current Lucene versions use BM25 as the default similarity, so nothing special is needed to turn it on. If you prefer to state the choice outright, add this line after creating the IndexWriterConfig:

config.setSimilarity(new BM25Similarity());

Ingestion code calls add() to put documents into Lucene. Its List<Document> parameter holds Spring AI documents (org.springframework.ai.document.Document), while Lucene indexes a type of its own (org.apache.lucene.document.Document). So the bulk of the method copies the ID, text and metadata of each Spring AI document into fields of a fresh Lucene document, which is then added to the index.

With the writer ready, declare it as a bean, inject it into the ApplicationRunner that performs ingestion, and write each batch of chunks to both stores:

@Bean
LuceneDocumentWriter luceneDocumentWriter(
    @Value("${lucene.index.path}") Path luceneIndexPath) throws IOException {
  return new LuceneDocumentWriter(luceneIndexPath);
}

@Bean
ApplicationRunner loadSpringBootDocumentation(
    VectorStore vectorStore, LuceneDocumentWriter luceneDocumentWriter) {
  return args -> {

    // ...code snipped out for brevity

    vectorStore.add(chunks);
    luceneDocumentWriter.add(chunks);

    // ...code snipped out for brevity

  };
}

The index path arrives through @Value, so set it in application.properties:

lucene.index.path=/tmp/lucene/boot-docs

Any path will do, but remember it, because the main RAG application has to open the same Lucene index.

If you worked through an earlier HyDE step in the same project, clear the existing vector store data before running ingestion again, or you'll store duplicate chunks. Once ingestion finishes you should again have 300 chunks across 89 documents, now reachable through both the vector store and Lucene.

Querying both sources

The main RAG application needs Lucene Core plus Lucene's query parser:

implementation 'org.apache.lucene:lucene-core:10.5.1'
implementation 'org.apache.lucene:lucene-queryparser:10.5.1'

Now create a component that searches the Lucene index and returns matches as Spring AI Document objects:

public class LuceneSearch implements AutoCloseable {

  private static final String ID_FIELD = "id";
  private static final String CONTENT_FIELD = "content";
  private static final String METADATA_FIELD = "metadata";

  private final Analyzer analyzer;
  private final Directory directory;
  private final DirectoryReader reader;
  private final IndexSearcher searcher;
  private final ObjectMapper objectMapper;

  public LuceneSearch(Path indexPath) throws IOException {
    this.analyzer = new StandardAnalyzer();
    this.directory = FSDirectory.open(indexPath);
    this.reader = DirectoryReader.open(directory);
    this.searcher = new IndexSearcher(reader);
    this.objectMapper = new ObjectMapper();
  }

  public List<Document> search(String queryText, int topK) {
    try {
      var parser = new QueryParser(CONTENT_FIELD, analyzer);
      var query = parser.parse(
          QueryParser.escape(queryText));

      var hits = searcher.search(query, topK);
      var storedFields = searcher.storedFields();

      var results = new ArrayList<Document>();
      for (var hit : hits.scoreDocs) {
        var luceneDocument =
            storedFields.document(hit.doc);

        var metadata = objectMapper.readValue(
            luceneDocument.get(METADATA_FIELD),
            new TypeReference<Map<String, Object>>() {
            });

        results.add(new Document(
            luceneDocument.get(ID_FIELD),
            luceneDocument.get(CONTENT_FIELD),
            metadata));
      }

      return results;
    }
    catch (IOException | ParseException e) {
      throw new IllegalStateException("Failed to search Lucene index", e);
    }
  }

  @Override
  public void close() throws IOException {
    reader.close();
    directory.close();
    analyzer.close();
  }
}

Without digging deep into Lucene's search API, search() takes the query text and asks an IndexSearcher for the best matching documents, with topK limiting how many come back. As in indexing, BM25 is the default similarity, and if you want to make that visible you can configure the searcher explicitly:

searcher.setSimilarity(new BM25Similarity());

The rest of search() converts each Lucene Document back into a Spring AI Document so it can take part in the RAG pipeline.

There are now two independent ways to retrieve documents. Vector similarity search through VectorStore is strong at finding semantically related content. BM25 search through LuceneSearch is strong at matching the important terms in the query. Either can give useful results alone, but combining them brings in both meaning and lexical precision. HybridDocumentRetriever handles that combination.

The hybrid document retriever

This is the core of it:

public class HybridDocumentRetriever implements DocumentRetriever {

  private static final Logger logger =
      LoggerFactory.getLogger(HybridDocumentRetriever.class);

  private final VectorStore vectorStore;
  private final LuceneSearch luceneSearch;

  public HybridDocumentRetriever(
      VectorStore vectorStore, LuceneSearch luceneSearch) {
    this.vectorStore = vectorStore;
    this.luceneSearch = luceneSearch;
  }

  @Override
  public List<Document> retrieve(Query query) {
    var vectorResults = vectorStore.similaritySearch(
        SearchRequest.builder()
            .query(query.text())
            .topK(10)
            .build());

    var bm25Results = luceneSearch.search(query.text(), 10);

    var fusedResults =
        reciprocalRankFusion(vectorResults, bm25Results, 10);

    logResults("VECTOR", vectorResults);
    logResults("BM25", bm25Results);
    logResults("FUSED", fusedResults);

    return fusedResults;
  }

  // ...

}

HybridDocumentRetriever implements Spring AI's DocumentRetriever, the component that fetches documents for RetrievalAugmentationAdvisor. Earlier steps used VectorStoreDocumentRetriever, which draws only from a vector store. Here it's replaced by a custom implementation.

Its retrieve() method runs two searches for one query. First it calls similaritySearch() for the top 10 vector hits. Then it asks Lucene for the top 10 BM25 hits. The result is two ranked lists, where some documents show up in both and others in just one. Joining the lists end to end would throw away ranking information, so the merge has to respect each document's position in each list. Reciprocal Rank Fusion does exactly that.

Merging with Reciprocal Rank Fusion

The fusion method looks like this:

private List<Document> reciprocalRankFusion(
    List<Document> vectorResults,
    List<Document> bm25Results,
    int limit) {

  Map<String, Double> scores = new HashMap<>();
  Map<String, Document> documents = new HashMap<>();

  addRrfScores(vectorResults, scores, documents);
  addRrfScores(bm25Results, scores, documents);

  return scores.entrySet().stream()
      .sorted(Map.Entry.<String, Double>comparingByValue().reversed())
      .limit(limit)
      .map(entry -> documents.get(entry.getKey()))
      .toList();
}

private void addRrfScores(
    List<Document> results,
    Map<String, Double> scores,
    Map<String, Document> documents) {

  final int k = 60;

  for (int i = 0; i < results.size(); i++) {
    var document = results.get(i);
    var rank = i + 1;

    documents.putIfAbsent(document.getId(), document);

    scores.merge(
        document.getId(),
        1.0 / (k + rank),
        Double::sum);
  }
}

Reciprocal Rank Fusion (RRF) turns several ranked result sets into one ranking. Instead of trying to compare raw scores from two very different retrieval algorithms, it scores documents by where they rank in each list. For every list, a document receives:

1 / (k + rank)

Here k is a constant, commonly 60 for RRF, while rank stands for where the document sits in that list. A document found in both lists gets the two scores added together. The outcome is intuitive: documents ranked highly by either search do well, and documents ranked highly by both tend to climb to the top, which is what you want when blending semantic and lexical retrieval.

The end of retrieve() logs all three rankings so the effect is visible:

private void logResults(String label, List<Document> documents) {
  logger.debug("{}", label);

for (int i = 0; i < documents.size(); i++) {
    var document = documents.get(i);

    var text = document.getText()
        .replaceAll("\\s+", " ")
        .trim();

    var snippet = text.substring(
        0, Math.min(text.length(), 80));

    logger.debug(
        "{}. {} - {}",
        i + 1,
        document.getId(),
        snippet);
  }
}

That logging isn't needed for hybrid search itself, but while experimenting it lets you compare the vector, BM25 and fused orderings side by side.

Wiring up hybrid retrieval

Register LuceneSearch and HybridDocumentRetriever as beans:

@Bean
LuceneSearch luceneSearch(
    @Value("${lucene.index.path}") Path luceneIndexPath) throws IOException {
  return new LuceneSearch(luceneIndexPath);
}

@Bean
HybridDocumentRetriever hybridDocumentRetriever(
    VectorStore vectorStore, LuceneSearch luceneSearch) {
  return new HybridDocumentRetriever(vectorStore, luceneSearch);
}

As with ingestion, LuceneSearch needs the location of the Lucene index:

lucene.index.path=/tmp/lucene/boot-docs

Make sure this matches the path the ingestion application used. Then change the ChatClientBuilderCustomizer that configures RetrievalAugmentationAdvisor so it uses HybridDocumentRetriever in place of VectorStoreDocumentRetriever:

@Bean
ChatClientBuilderCustomizer addRagAdvisor(
    QueryTransformer[] queryTransformers,
    HybridDocumentRetriever hybridDocumentRetriever) {
  return builder -> {
    var ragAdvisor = RetrievalAugmentationAdvisor.builder()
        .queryTransformers(queryTransformers)
        .documentRetriever(hybridDocumentRetriever)
        .documentJoiner(documentsForQuery ->
            documentsForQuery.values().stream()
                .flatMap(List::stream)
                .flatMap(List::stream)
                .toList())
        .build();

    builder.defaultAdvisors(ragAdvisor);
  };
}

Besides passing the retriever through documentRetriever(), the customizer also hands documentJoiner() a lambda-based DocumentJoiner. A joiner's job is to merge several lists of retrieved documents into one collection. The default ConcatenationDocumentJoiner re-sorts by each document's score, which would undo the ordering RRF just produced. So the code supplies an essentially do-nothing joiner that leaves the fused ranking alone.

You might think the joiner is the natural place for RRF, and conceptually it is. That design would also suggest splitting vector search and Lucene search into two separate retrievers. The catch is that the advisor's builder accepts only one DocumentRetriever, so the two mechanisms can't be configured independently and both result sets handed to the joiner. Doing both searches and the fusion inside HybridDocumentRetriever, then using a pass-through joiner to keep the fused order, is the simpler route. If Spring AI later supports multiple retrievers, the design is worth revisiting.

That is all the wiring. RetrievalAugmentationAdvisor doesn't have to know two search strategies are in play. To it, HybridDocumentRetriever is just the component that supplies the documents added to the prompt context.

Seeing the difference

Turn on debug logging for HybridDocumentRetriever so you can compare the three rankings:

logging.level.com.example.essentialrag.HybridDocumentRetriever=debug

Start the application and pose a question about using @ConfigurationProperties to map outside configuration onto a Java object with validation. The debug output will show three result sets labelled VECTOR, BM25 and FUSED. The vector ranking begins like this:

VECTOR
1. d91bc5c8-... - If the annotation does not provide any prefix...
2. 049f6484-... - For YAML, you can use the following syntax...
3. 152dde59-... - In this setup, the presence of a single parameterized constructor...
...

BM25 produces a different order:

BM25
1. 4c7128dc-... - There are spring-boot-starter-data-couchbase...
2. a3dbefae-... - fun main(args: Array<String>)...
3. 1ab4c223-... - Configuring System Environment Properties...
4. d91bc5c8-... - If the annotation does not provide any prefix...
...

And the RRF output blends them:

FUSED
1. d91bc5c8-... - If the annotation does not provide any prefix...
2. 1ab4c223-... - Configuring System Environment Properties...
3. 049f6484-... - For YAML, you can use the following syntax...
4. 152dde59-... - In this setup, the presence of a single parameterized constructor...
5. 091acd83-... - The default unit is bytes...
...

The contrast shows why hybrid search earns its keep. Vector search handles the conceptual parts of the question well: external configuration, binding, Java objects and validation. BM25 adds lexical matching and puts weight on distinctive terms such as ConfigurationProperties. RRF then merges the two views. Documents that score well in both climb toward the top, while strong hits found by only one strategy can still reach the final list.

The pipeline therefore doesn't have to choose between grasping what the user means and matching the words they used. With hybrid search it can do both.