Home / Articles

Conversation-Aware RAG in Spring AI with CompressionQueryTransformer

Fix vague follow-up questions in a Spring AI RAG chat by rewriting queries with chat history before they reach the vector store, using two ChatClient builders.

Publisher: REACTAPP.TOP6 min read1455 words

This article is published in English.

Sources: Spring AI Recipe: Making RAG Conversation-Aware

A RAG chatbot works best when retrieval understands the conversation and not only the latest message. Spring AI can fold relevant chat history into the query sent to the vector store, so follow-up questions pull back the right context.

The follow-up problem

Picture a RAG chat app built over the 2026 NFL Rulebook. The first question asks for a definition of a hip-drop tackle. QuestionAnswerAdvisor passes that very text to the vector store, the closest chunks come back and get appended to the prompt, and the LLM writes the definition. So far nothing is wrong.

The next message is just "Is there a penalty?" A person knows the topic is still the hip-drop tackle. The vector store doesn't. Before the LLM sees anything, the store receives only those four words. If it's lucky it returns something on hip-drop tackles. More likely it returns penalties for unrelated plays. Conversational RAG has this weakness in general: people ask follow-ups without restating context, and the retrieval step has no memory of earlier turns.

Reproducing it

Start from the existing recipe for RAG without a separate vector database, and point the rag.document.url property in application.properties at the rulebook PDF, which the NFL publishes on its operations site:

rag.document.url=file:///Users/habuma/Documents/ops-2026-NFL-Rulebook-FINAL.pdf

Then run the application unchanged and ask about hip-drop tackles. A session looks like this:

How can I help?

> What is a hip drop tackle?

 - A hip-drop tackle is a foul defined as when a player (a) grabs or wraps the
runner with one or both arms and then (b) unweights himself by swiveling and
dropping his hips and/or lower body, landing on and/or trapping the runner's
leg(s) at or below the knee. Penalty: Loss of 15 yards and an automatic first
down.

> Is there a penalty?

 - I can't answer that from the provided context. The excerpt you gave
contains general penalties for unsportsmanlike conduct, personal fouls,
enforcement spots, and related rules (many unsportsmanlike/personal-foul
penalties are 15 yards, sometimes with disqualification or an automatic first
down), but it does not define a "hip drop tackle" or state a specific penalty
for it.

The second question isn't answered from the retrieved context. The store never searched for anything about hip-drop tackles, only for the vague penalty question. The reply does mention the tackle, which shows the chat history kept the context. The problem is that retrieval never saw that history. What's needed is a way to include relevant history in the vector store query, and Spring AI provides one.

A different advisor

QuestionAnswerAdvisor isn't the only RAG advisor in Spring AI. RetrievalAugmentationAdvisor is more flexible and lets you customize almost every retrieval stage. One extension point is a query transformer, which rewrites the user's query before it reaches the store.

This recipe uses CompressionQueryTransformer. Despite the name, it doesn't merely shorten the query. It rewrites it into a compact, self-contained form that carries in whatever matters from the conversation so far. The vector store then gets a far more meaningful query.

The transformer is declared as a bean:

@Bean
QueryTransformer queryTransformer(
    @Qualifier("queryCompressionChatClientBuilder") ChatClient.Builder chatClientBuilder) {
  return CompressionQueryTransformer.builder()
      .chatClientBuilder(chatClientBuilder)
      .build();
}

It asks an LLM to build a better query from the current question and the chat history, which is why a ChatClient.Builder is injected. That injected builder carries a @Qualifier. Reusing the builder from the chat loop would create a circular dependency, and keeping them apart also allows a different model for compression than for answering.

Here OpenAI serves both builders, so no extra provider like Anthropic or Ollama is needed, but two separate ChatClient.Builder beans are defined:

@Bean
@Primary
ChatClient.Builder openAiChatClientBuilder(
    OpenAiChatModel openAiChatModel,
    ObjectProvider<ChatClientBuilderCustomizer> customizers) {

  return applyCustomizers(
      ChatClient.builder(openAiChatModel),
      customizers
  );
}

@Bean
@Qualifier("queryCompressionChatClientBuilder")
ChatClient.Builder compressionChatClientBuilder(
    OpenAiChatModel openAiChatModel) {
  return ChatClient.builder(openAiChatModel)
      .defaultOptions(OpenAiChatOptions.builder()
          .model("gpt-5-nano"));
}

private ChatClient.Builder applyCustomizers(
    ChatClient.Builder builder,
    ObjectProvider<ChatClientBuilderCustomizer> customizers) {
  customizers.orderedStream()
      .forEach(customizer -> customizer.customize(builder));
  return builder;
}

The setup resembles earlier recipes with multiple models. The first builder replaces Spring AI's auto-configured one so registered ChatClientBuilderCustomizers still apply, which happens by passing the builder and customizers to the applyCustomizers() helper. It's marked @Primary, making it the default unless a @Qualifier selects another. The second builder exists for query compression. Rewriting queries is cheap work, so a small, low-cost model such as gpt-5-nano fits well, though any model can be used.

With a transformer and its own builder in place, QuestionAnswerAdvisor can give way to RetrievalAugmentationAdvisor, configured through a ChatClientBuilderCustomizer:

@Bean
ChatClientBuilderCustomizer addRagAdvisor(
    QueryTransformer queryTransformer,
    VectorStore vectorStore) {

  return builder -> {
    var ragAdvisor = RetrievalAugmentationAdvisor.builder()
        .documentRetriever(VectorStoreDocumentRetriever.builder()
            .vectorStore(vectorStore)
            .build())
        .queryTransformers(queryTransformer)
        .build();
    builder.defaultAdvisors(ragAdvisor);
  };
}

This advisor takes a little more setup but offers more control. It first needs a document retriever, here a VectorStoreDocumentRetriever pulling matches from the injected VectorStore. The important part is the queryTransformers() call, which receives the CompressionQueryTransformer bean. From then on, every query bound for the store is first rewritten to include relevant conversation context, giving retrieval a far better chance of finding the right documents.

The same conversation again

Repeating the dialogue gives a different result:

How can I help?

> What is a hip drop tackle?

 - A hip-drop tackle is when a player (a) grabs the runner with one or both
hands or wraps the runner with both arms, and (b) unweights himself by
swiveling and dropping his hips and/or lower body, landing on and/or trapping
the runner's leg(s) at or below the knee. Penalty: Loss of 15 yards and an
automatic first down.

> Is there a penalty?

 - Yes. Loss of 15 yards and an automatic first down (the player may be
disqualified if the action is flagrant).

The app explains the tackle and also answers the follow-up correctly, though the user never mentions hip-drop tackles a second time.

The difference sits in the query that reached the vector store. Spring AI doesn't log the rewritten text at the moment, so the quickest way to look at it is a breakpoint on the return line inside CompressionQueryTransformer.transform(). Two rewrites show up there:

  • The first question became a longer one asking what a hip-drop tackle is, which sport uses it, how it's performed and what its legality and safety implications are.
  • The vague penalty question became a complete one asking whether American football penalizes a hip-drop tackle, and if so how many yards and whether an automatic first down follows.

The second rewrite explains the improvement. In place of an ambiguous question, the store got a full, standalone one that included context from the history, so it returned passages about hip-drop tackles and not penalties for other plays.

Interestingly, the first query was rewritten too. It didn't depend on earlier turns, yet the transformer expanded it into a more descriptive question, which can improve retrieval by making intent clearer.

That's the value of CompressionQueryTransformer. Users don't have to restate everything with each follow-up. They can talk naturally while the store receives the rich, explicit queries it needs.

It's also only one example of what RetrievalAugmentationAdvisor allows. Query transformation is one extension point in the pipeline. Document retrieval, post-processing of retrieved documents, query expansion and other stages can be customized as well.

Trade-offs to keep in mind

Query compression isn't free. Each user turn now triggers an extra LLM call before retrieval even starts, which adds latency and a little cost. That's the reason for giving the transformer its own small model: the rewriting task is short and well-defined, so a compact model usually handles it, while the main model stays reserved for the final answer.

Since the rewritten query isn't logged by default, it's worth adding your own logging or a temporary breakpoint while tuning. Reading a handful of original and rewritten pairs quickly shows whether the transformer is dropping details or padding queries with guesses, such as the sport name in the first example above. If retrieval quality looks off, the rewritten text is the first place to look.