Spring AI Basics: ChatClient, Streaming, Memory and Tools With Ollama

Imagine an AI feature built on OpenAI today. Next quarter the bill grows, the team wants to try Ollama on laptops, or the company standardizes on Azure OpenAI, Bedrock or Gemini. With no abstraction layer, changing providers can ripple through the codebase, touching client setup, request shapes, streaming code, embeddings, tool definitions and the way replies get parsed. Spring AI gives Spring Boot developers one set of APIs across providers, which keeps that ripple small.

The focus here is practical basics: what the framework offers, project setup, Ollama configuration, ChatClient, prompt templates, streaming, structured output, chat memory and tool calling. Retrieval, vector stores, ETL, filtering, observability, evaluation, multimodal input, MCP and production advice are left for a follow-up topic.

Every example runs on Ollama, so nobody needs an API key to follow along. The sample project is built with Spring Boot 3.5.5 and Spring AI 1.0.9. A stable Spring AI 2.0.x line exists too, though it is aimed at Spring Boot 4.0.x and 4.1.x, which means the two version sets shouldn't be mixed in one build.

Vocabulary before code

For newcomers, the terminology can be harder than the code. These are the terms used below.

• LLM (large language model): the model that generates text. GPT, Claude, Gemini and local models such as Llama all qualify. Text goes in, and the model predicts useful text coming out.
• Model: the concrete engine being called. Here Ollama runs it locally. In the cloud it could sit on OpenAI, Anthropic, Azure OpenAI, Amazon Bedrock or Google Vertex AI.
• Prompt: the instruction or question sent to the model, for example "Explain Spring AI in one paragraph." Clear, specific prompts with enough context get better answers.
• Token: a small unit of text, which may be a word, part of a word, punctuation or whitespace. Pricing, context size and output length are normally counted in tokens.
• Context window: the maximum text the model can consider at once. Prompt, chat history, retrieved documents, tool results and the reply all count against it.
• ChatClient: Spring AI's fluent API for chat models. Rather than hand-building provider-specific HTTP calls, code reads like chatClient.prompt().user(...).call().content().
• Prompt template: a reusable prompt with placeholders. "Explain {topic} in {language}" turns into "Explain RAG in Java" at runtime.
• Structured output: asking the model for data in a predictable shape and mapping it onto a Java record or class. Useful when an application needs fields rather than prose.
• Chat memory: conversation history that the application manages. Models are stateless by default, so earlier messages must be resent whenever the model should recall them.
• Advisor: a Spring AI component that alters or enriches a request before it reaches the model. A memory advisor, for instance, adds earlier messages, and a RAG advisor, covered later on, would add retrieved documents.
• Tool calling: the model asks the application to run a Java method. It never touches a database or API itself. It requests the tool, the application executes it, and the result goes back to the model.

What Spring AI includes

The framework is more than a wrapper around chat completions. For Java teams it works as a full application framework for AI workloads. Its main pieces:

• portable model APIs spanning chat, embeddings, images, speech-to-text, text-to-speech and moderation
• integrations for OpenAI, Anthropic, Azure OpenAI, Amazon Bedrock, Google Vertex AI and Gemini, Mistral, Ollama and more
• ChatClient, a fluent API in the spirit of WebClient and RestClient
• prompt templates and message types for system, user, assistant and media content
• mapping of model replies into Java records or classes
• streaming through Project Reactor
• advisors for recurring patterns like memory and RAG
• tool calling, so Java methods can be requested by the model
• embedding models behind vector store abstractions
• extras aimed at production: ETL, observability, evaluation and MCP

That's a wide surface, so the walkthrough below keeps to the core application flow.

Project setup

Create a Maven project on Java 21 with Spring Boot 3.5.x, Spring Web and the Spring AI Ollama starter. A bill of materials keeps every Spring AI module on compatible versions.

<properties>
    <java.version>21</java.version>
    <spring-ai.version>1.0.9</spring-ai.version>
</properties>
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-bom</artifactId>
            <version>${spring-ai.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

For the local Ollama demo, add the web starter along with the Ollama one.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-ollama</artifactId>
</dependency>

Anyone preferring OpenAI can swap the model starter.

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>

One warning applies: keep a single chat model starter on the classpath unless a primary ChatModel bean is configured. When OpenAI and Ollama are both there, Spring finds two candidates and can't build the default ChatClient.Builder, so startup fails.

Running a local model

Install Ollama and pull a chat model.

ollama pull llama3.2

Then point application.properties at it.

spring.application.name=spring-boot-ollama
spring.ai.ollama.base-url=http://localhost:11434
spring.ai.ollama.chat.options.model=llama3.2

Start the app.

mvn spring-boot:run

It listens here:

http://localhost:8080

Basic chat with ChatClient

ChatClient is the high-level API that most Spring applications should begin with.

@RestController
public class SpringAiDemoController {

    private final ChatClient chatClient;

    public SpringAiDemoController(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    @GetMapping("/api/demo/simple")
    public String simpleChat(@RequestParam(defaultValue = "Tell me a joke") String message) {
        return chatClient.prompt()
                .user(message)
                .call()
                .content();
    }
}

Trying it from a terminal:

curl "http://localhost:8080/api/demo/simple?message=Explain%20Spring%20AI%20in%20one%20paragraph"

The call unfolds in four steps. prompt() opens a new request, user(...) attaches the message, call() chooses blocking execution, and content() returns the generated text.

Prompt templates

Hardcoded prompts get painful to maintain. A template holds reusable text with parameters.

@GetMapping("/api/demo/template")
public String templateChat(
        @RequestParam(defaultValue = "Microservices") String topic,
        @RequestParam(defaultValue = "Java") String language) {

    PromptTemplate template = new PromptTemplate(
            "Explain the concept of {topic} using examples in {language}. Keep it brief.");

    String renderedPrompt = template.render(Map.of(
            "topic", topic,
            "language", language
    ));

    return chatClient.prompt()
            .user(renderedPrompt)
            .call()
            .content();
}

Calling the endpoint with a topic and a language:

curl "http://localhost:8080/api/demo/template?topic=Retrieval-Augmented%20Generation&language=Java"

Templates fit best when the prompt is part of the application's behavior rather than a one-off string.

Streaming

Long responses should stream, so users see output right away instead of waiting for the full text.

@GetMapping(value = "/api/demo/stream", produces = "text/event-stream")
public Flux<String> streamChat(
        @RequestParam(defaultValue = "Write a short story about a brave robot.") String message) {return chatClient.prompt()
            .user(message)
            .stream()
            .content();
}

Test it in a browser or with curl in no-buffer mode.

curl -N "http://localhost:8080/api/demo/stream?message=Write%20a%20short%20Spring%20AI%20intro"

Spring MVC writes the Flux of strings out as Server-Sent Events.

Structured output

Text suits chat, but business software usually needs structured data. Spring AI can request a response that maps onto a Java type. With local models, spelling out the JSON format explicitly tends to work better.

public record MovieReview(String title, double rating, String shortSummary) {
}

@GetMapping("/api/demo/structured")
public MovieReview structuredOutput(@RequestParam(defaultValue = "Inception") String movie) {
    return chatClient.prompt()
            .user(u -> u.text("Review the movie {movie}.")
                    .param("movie", movie))
            .call()
            .entity(MovieReview.class);
}

Try it:

curl "http://localhost:8080/api/demo/structured?movie=The%20Matrix"

Calling entity with the record class makes Spring AI add format instructions to the prompt and convert the returned JSON straight into the record. That's tidier and more dependable than parsing JSON strings by hand.

Conversation memory

Models don't recall earlier requests, so the application supplies the history. Spring AI handles that with memory abstractions and advisors, registered as defaults on the client builder.

this.chatClient = chatClientBuilder
        .defaultAdvisors(
                MessageChatMemoryAdvisor.builder(
                        MessageWindowChatMemory.builder().build()
                ).build()
        )
        .build();

Each request then carries a conversation id.

@GetMapping("/api/demo/memory")
public String chatWithMemory(
        @RequestParam String message,
        @RequestParam(defaultValue = "chat_session_1") String chatId) {
        return chatClient.prompt()
            .user(message)
            .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, chatId))
            .call()
            .content();
}

Two calls sharing an id show the effect: first state a name, then ask what it was.

curl "http://localhost:8080/api/demo/memory?chatId=demo&message=My%20name%20is%20Yash"
curl "http://localhost:8080/api/demo/memory?chatId=demo&message=What%20is%20my%20name?"

In production, replace in-memory storage with a persistent repository such as JDBC, Redis, Cassandra, MongoDB or Neo4j.

Tool calling

Tools let the model request work defined by the application. The model has no direct path to a database or an API. It asks, application code runs the tool, and the result returns to the model.

A tool is a plain method with descriptions on it.

public static class WeatherTools {
@Tool(description = "Get the current weather for a city")
    public WeatherResponse getCurrentWeather(
            @ToolParam(description = "City name") String city) {
        return new WeatherResponse(city, 22.5, "Sunny");
    }
}
public record WeatherResponse(
        String location,
        double temperatureCelsius,
        String condition) {
}

It gets exposed as a bean.

@Bean
public WeatherTools weatherTools() {
    return new WeatherTools();
}

Then a request enables it.

@GetMapping("/api/demo/function")
public String functionCalling(
        @RequestParam(defaultValue = "What is the weather in Paris?") String message) {
        return chatClient.prompt()
            .user(message)
            .tools(weatherTools)
            .call()
            .content();
}

Asking about the weather in a city exercises the full loop.

curl "http://localhost:8080/api/demo/function?message=What%20is%20the%20weather%20in%20Paris?"

Good uses include checking inventory, fetching weather, looking up a customer, opening a support ticket or kicking off a workflow. Destructive actions call for strictness: add authorization, validation, auditing and user confirmation where needed.

Running the whole demo

Start the application again.

mvn spring-boot:run

Then exercise the endpoints in turn.

curl "http://localhost:8080/api/demo/simple?message=Explain%20Spring%20AI%20in%20one%20paragraph"
curl "http://localhost:8080/api/demo/template?topic=Retrieval-Augmented%20Generation&language=Java"
curl -N "http://localhost:8080/api/demo/stream?message=Write%20a%20short%20Spring%20AI%20intro"
curl "http://localhost:8080/api/demo/structured?movie=Inception"
curl "http://localhost:8080/api/demo/memory?chatId=demo&message=My%20name%20is%20Yash"
curl "http://localhost:8080/api/demo/memory?chatId=demo&message=What%20is%20my%20name?"
curl "http://localhost:8080/api/demo/function?message=What%20is%20the%20weather%20in%20Paris?"

Mistakes that come up often

Adding several model starters without choosing one produces this startup error:

No qualifying bean of type 'ChatModel' available: expected single matching bean but found 2

Keep a single starter in the demo or name a primary model explicitly.

Using chat memory without a conversation id means every call looks like a new chat. Pass the id as an advisor parameter.

.advisors(a -> a.param(ChatMemory.CONVERSATION_ID, chatId))

Exposing tools too freely is the third trap. Tools are real application actions, so validate their inputs, enforce authorization, and avoid destructive operations unless an approval flow exists.

Where things stand

At this point the foundation of a Spring AI application is in place: chat, prompts, streaming, structured output, memory and tools. Those pieces already cover plenty of useful scenarios, including chat assistants, summarizers, structured-extraction endpoints and controlled tool-calling flows. What comes after chat is a knowledge layer built on embeddings and vector stores, with RAG, ETL pipelines, metadata filters, observability, evaluation, multimodal input, MCP and production hardening on top.

A complete demo project can be cloned from GitHub.

git clone https://github.com/Yash2462/blog-demos.git

After cloning, open the spring-ai-demo folder and follow the README there.

Further reading

• the Spring AI reference documentation
• the Spring AI getting started guide
• the Ollama chat model page in the Spring AI docs
• the ChatClient documentation
• the tool calling documentation