Java as an AI Production Language: Spring Boot, RAG and pgvector

Python trains the models, and Java is quietly turning into the language that gets them shipped. For roughly three years the AI scene felt like a Python conference with a bigger budget: demos in notebooks, tutorials that opened with pip install, and architecture diagrams where a little box called "Python LLM service" hung off the real backend. That real backend was usually Spring Boot, already in charge of auth, billing, audit logs and the database.

The little box isn't mandatory anymore. Spring AI is past 1.0 and now at 2.0. LangChain4j is a proper JVM library with over twenty model providers and over thirty vector stores. PostgreSQL stores embeddings through pgvector. A Spring Boot service you already run can call an LLM, fetch grounded context, invoke tools, stream tokens and publish Micrometer metrics, all inside the JVM.

Does that make Java the new AI king? Not in the sense of the usual language flame wars. Nobody is swapping PyTorch for Java. What Java is winning is the ground where the money is: business applications that must be correct, observable and dull in the best way. If you work there, the winner is whatever already runs your production cluster, and it increasingly speaks Java.

The rest of this piece maps the shift. It covers why it happened, what a RAG service in Spring Boot looks like, how to choose a vector store, and which production details separate a demo from something you'd trust with customer data.

Python's edge was always research

Python took the research era for good reasons that still apply: NumPy, PyTorch, Hugging Face, Jupyter, and a whole generation of engineers fluent in tensors. Fine-tuning a model, testing a new embedding architecture or training on GPUs still belongs in Python.

Shipping is another game. Most firms never train a foundation model. They call one, and then the real work starts: fetching documents, injecting policy, separating tenants, logging prompts, capping spend, and making sure nothing invents a refund figure. That's API design, data access, security and operations, which Spring Boot has handled since 2014.

The popular shortcut of putting the model behind FastAPI carried a recurring cost:

• two languages inside a single product
• separate deployment pipelines
• separate monitoring stacks
• an interface contract that wandered a little each sprint
• security reviews that suddenly included a Python service holding its own secrets

People accepted this because Java lacked serious LLM tooling. By 2025 that stopped being true. Spring AI 1.0 became generally available in May 2025, and version 2.0 arrived in June 2026 alongside Spring Boot 4. LangChain4j reached 1.0 in the same period. The JVM stack isn't playing catch-up now. For teams already in Spring it's a sensible default.

What "adding AI" means in a Spring app

There's no rewrite involved, just three capabilities.

1. A chat layer. ChatClient in Spring AI, or an AiService in LangChain4j, converts a prompt and message history into a model reply. Streaming comes standard, and structured output can land in a Java record, which saves you from regex-ing JSON out of text.
2. Access to your own data. You compute embeddings and keep them in a vector store, so answers come from your policies, invoices, runbooks and product documentation, not from whatever the model half-remembers. This is Retrieval-Augmented Generation (RAG).
3. Tool calls. The model may invoke Java methods under your authorization rules, for example to look up an order, open a case or read a balance. Here the type system and the services you already own pay off. The model isn't your source of truth. It reasons on top of one.

With Spring Security, Spring Data, Actuator and PostgreSQL in place, this is one more client type, much like when Redis or Kafka arrived.

RAG, described for a Java developer

Strip away the mystique and RAG is a search pipeline that ends in a language model. The steps:

1. Cut documents into chunks.
2. Turn each chunk into an embedding, a vector of floats that encodes its meaning.
3. Save the vectors with metadata such as tenant, document id, source URI and as-of date.
4. When a question arrives, embed it and look up the closest chunks.
5. Insert those chunks into the prompt.
6. Instruct the model to answer from them alone, and to admit when they don't contain the answer.

A vector database isn't a substitute for your transactional one. It's a semantic index, a bit like Elasticsearch with fancier vocabulary.

For Spring Boot shops, PostgreSQL plus pgvector is normally where you begin. Postgres is already deployed, backed up and pooled, so one extension costs less than bringing in Pinecone, Weaviate or Qdrant immediately. Dedicated engines do win when scale is extreme or you need hybrid search, yet most internal RAG apps stay well below that point.

A skeleton that runs

The setup below pairs Spring AI 2.0 with Boot 4; on Boot 3.4 or 3.5 use the 1.1.x line instead. Bring in the BOM first so every starter lines up on the same version:

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.springframework.ai</groupId>
      <artifactId>spring-ai-bom</artifactId>
      <version>1.1.8</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

Next, declare just the dependencies you need:

<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-starter-vector-store-pgvector</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>

JDBC is listed on purpose. The pgvector starter works through JdbcTemplate, and forgetting that dependency is a surprisingly frequent cause of a failed boot.

Settings go in application.yml. That's the point of Spring AI: a model should be configured like a datasource.

spring:
  datasource:
    url: jdbc:postgresql://localhost:5432/app
    username: app
    password: ${DB_PASSWORD}
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      chat:
        options:
          model: gpt-4o-mini
          temperature: 0.2
      embedding:
        options:
          model: text-embedding-3-small
    vectorstore:
      pgvector:
        initialize-schema: true
        index-type: HNSW
        distance-type: COSINE_DISTANCE
        dimensions: 1536

Two details in this file matter more than they look. The embedding dimension must equal what the embedding model produces, and text-embedding-3-small produces 1536. Change the model, forget the number, and similarity search degrades without any error. The second is a low temperature for RAG, because you want output hugging the retrieved text, not a creative rewrite of a balance sheet.

Treat ingestion as its own service instead of a controller one-liner. VectorStore.add(...) accepts Document chunks, and each chunk should carry the metadata you'll filter on later, such as tenantId, source, docType and effectiveDate. Without filters, semantic search can surface one client's contract in another client's answer.

Querying is just as short. The QuestionAnswerAdvisor in Spring AI is an opinionated version of the loop: retrieve, add context, call the model. Once you require filters, reranking or citations, use similaritySearch yourself, and read filterExpression as a WHERE clause, since that's exactly its role.

Choosing a vector store without dogma

You'll meet four options:

• Postgres with pgvector. It's the sensible default and operationally well understood. With HNSW indexing, recall monitoring and compact chunks, it copes with tens of millions of vectors. It falls short when you need advanced hybrid search or a team dedicated to running approximate nearest-neighbor infrastructure.
• Redis. Quick, often present in Spring stacks already, and a fit for session memory and small knowledge bases. RAM cost is what you end up debating.
• Purpose-built engines (Qdrant, Weaviate, Milvus, Pinecone). They make sense when retrieval is the product itself: big corpora, lexical-plus-vector hybrid queries, several indexes, or a team wanting software designed for ANN. Spring AI supports the popular ones directly.
• The search engine you already have. Elasticsearch and OpenSearch handle kNN, and if your documents live there, a dense vector field can beat adding a fourth data system.

The method for deciding is plain, and that's its strength. Begin on pgvector when Postgres already runs in production. Change course when recall, latency or hybrid search fail a real SLO, and not because a talk claimed that serious AI apps use some product. Also keep a reference to the original record next to each vector, so you can re-embed whenever a policy shifts.

That last point is what tutorials leave out. Embedding models get replaced, chunking gets smarter and legal documents get amended. An ingest job that's idempotent and keyed on document version spares you a quarter of chasing why the bot quotes last year's fee schedule.

Spring AI versus LangChain4j

Both are ready for production. Which one fits depends on where your team sits today.

Spring AI takes the familiar Spring idea of portable abstractions and applies it to models. If you picture it as Spring Data for LLMs, it'll feel natural, and auto-configuration, ChatClient, advisors, Actuator and Micrometer all slot in. Use 2.0 on Boot 4. On Boot 3, stay on 1.1.x, since 2.0 won't just drop in.

LangChain4j is written for Java first and offers a Spring starter on the side. It supports more providers, its typed AiServices resemble Feign clients pointed at models, and it suits teams that also use Quarkus or plain Java. People who want close control over the agent loop tend to choose it.

Nothing stops you from running both. One workable split is Spring AI for chat and RAG in the customer-facing service, with LangChain4j filling gaps for a particular provider or agent pattern. Keep their beans separate, and never let two different embedding models write incompatible vectors into one table.

An all-Spring-Boot estate that can schedule a Boot 4 upgrade should go with Spring AI. A mixed JVM environment, or one stuck on Boot 3 for a year, will probably find LangChain4j less dependent on the framework's release train.

Production is where the JVM pulls ahead

Anyone can demo an answer. A product has to withstand an audit. These are the areas to cover:

• Grounding and refusals. Limit the model to retrieved context and return citations. When similarity scores fall under a threshold, the reply should be "that isn't in the knowledge base" and not a guess. For finance, operations and healthcare, a confident error costs more than a flat refusal.
• Tenancy and redaction. Filter vectors per tenant and scrub secrets from prompts. Keep card numbers, account numbers and health data out of raw prompt logs. If debugging forces you to log them, tokenize first and retain under the source system's retention rules.
• Budgets for cost and latency. Tokens cost money, so record input tokens, output tokens, retrieval time and model time. Put a max-tokens limit on every call, cache embeddings for documents that repeat, and send classification or routing to a small model, using a larger one only when needed.
• Timeouts and isolation. A model call is remote I/O with a long tail. Bounded thread pools, circuit breakers and a degraded path (keyword search or a human queue) stop a model outage from eating all Tomcat or Netty threads and dragging checkout down with it.
• Evaluation. A prompt can't be unit-tested like a tax calculation, but a frozen golden set of thirty questions with known answers can break the build when numeric accuracy slips. Score retrieval for RAG as well, by checking whether the right chunk even entered the prompt. A lot of supposed model-quality problems are retrieval bugs.
• Human review. In enterprise software the most valuable AI feature is frequently a draft, not a chatbot: a suggested note on a case, a prefilled reason for an exception, or a digest of a long 40-page statement. Keep a person in the approval step until errors become uneventful.

All of this is familiar Spring work: wrap a remote call, tag a metric, reject unauthenticated requests. The LLM is a new dependency with a worse SLA, so manage it like one.

Where the "Java is king" line is true

Java is the better starting point when the system of record already lives on the JVM, when the AI feature has to share identity, tenancy and audit with existing APIs, when it needs to call internal Java services as tools, when compliance doesn't allow a shadow Python stack, and when the people on call at 2 a.m. already know Spring Boot.

Python still leads when you're training or fine-tuning, when you need a research library the week it's released, or when the model itself is the product and not one feature within a bigger application.

Those are separate jobs, and for two years the industry acted as if they weren't. The result was a flood of architectures with a Python sidecar and hope that its contract would hold still. The pattern that looks interesting for 2026 is mixed: Python, or a hosted model lab, handles training and evaluation, Java serves, vector search sits beside the operational data, the model is a vendor and the domain logic stays in your hands.

A sensible first project

Don't open with an unbounded company-wide chatbot. Pick a narrow, valuable body of content and a question you can score. Some that work:

• Runbook Q&A for on-call engineers, linked to the source page
• A policy assistant that quotes the exact clause
• Q&A over an invoice or statement a user has just uploaded
• Ticket summaries written into the case record, not into a separate chat silo

Put it behind an authenticated API you already have, log every retrieval, display citations in the UI, add a config kill switch, and find out the cost per request before enlarging the corpus. The first project teaches chunking, metadata, evaluation and prompt change control, and those skills carry over, while the framework brand doesn't.

The runtime you already trust

"Java is the new AI king" works as a headline and falls short as a thesis. Java didn't win machine learning. It pulled applied AI into the platform that already carries the world's transaction systems.

A Spring Boot engineer doesn't need anyone's permission to work here, or a data science degree first. What's required is a model provider, an embedding model, a vector index and the same operational care applied to payments and messaging. Python keeps training the stars of the lab. Java is becoming the leader on the road from lab to ledger, where answers need grounding, tool calls need authorization and every bill needs an explanation. For most teams that's the only crown worth having.

If a Spring Boot service of yours already controls identity, money and the audit trail, there's no reason to wait for a sidecar. Choose one narrow corpus this week, put ChatClient in front of pgvector, and require citations from the model.