This article is published in English.
Diagnosing CI Build Failures With Rules, RAG and a Local LLM
A Java system for CI failure analysis that parses logs deterministically, retrieves troubleshooting notes from PGVector with Ollama, then asks an LLM to reason.
A CI server is excellent at one thing: announcing that a build broke. What it rarely explains is the part engineers care about once a production pipeline goes red. Why did it break? Where's the actual defect? What needs to change? And could earlier engineering knowledge sharpen the diagnosis?
A failure analyzer built around those questions is the subject here. Pasting a raw log into an LLM and asking what went wrong isn't the plan. The design mixes deterministic engineering logic, semantic search, retrieval-augmented generation (RAG) and model reasoning. Development is incremental: log analysis first, then repository-aware agents, then automated fixes.
What the system produces
Given a failed build, the analyzer takes the log through a pipeline, sketched here at a high level.
CI/CD Build
|
v
Build Log
|
v
Log Preprocessing
|
v
Deterministic Analysis
|
+------------------+
| |
v v
Structured Data RAG
| |
| v
| Relevant Knowledge
| |
+--------+---------+
|
v
LLM Reasoning
|
v
Build Diagnosis
The output is structured. It names the failure type, the failing component, the source location, the stack trace, the root cause, a recommended fix and a confidence value.
Facts first, model second
The earliest architectural decision was about ordering. The naive version hands the log straight to the model.
Build Log
|
v
LLM
|
v
Root Cause
Its flaw is big. The model must work everything out from a log that may be full of noise. The analyzer instead pulls out whatever can be established with certainty. Consider a log containing a build-failure marker, a null-pointer exception raised while validating a payment, and a stack frame pointing into a payment service class.
BUILD FAILURE
java.lang.NullPointerException:
Cannot invoke "PaymentValidator.validate()"
because "validator" is null
at com.example.payment.PaymentService.process(
PaymentService.java:47)
A deterministic parser can read off the status, the error category, and the exact file and line.
{
"status": "FAILED",
"errorType": "RUNTIME_ERROR",
"sourceLocation": "PaymentService.java:47",
"stackTrace": "..."
}
Nothing there involves guessing; it's all lifted from the log. With those facts handed over, the LLM can concentrate on interpreting them instead of doing basic parsing. That leads to a guiding formula.
Deterministic Analysis
+
RAG Knowledge
+
LLM Reasoning
=
Context-Aware Diagnosis
Adding retrieval
The next question was whether relevant engineering knowledge could be fetched before the model is asked to diagnose anything. That is where RAG comes in. A small knowledge base of troubleshooting notes was written, one Markdown file per topic: null-pointer exceptions, illegal-argument exceptions, dependency injection in Spring, and so on.
knowledge/
├── null-pointer-exceptions.md
├── illegal-argument-exceptions.md
├── spring-dependency-injection.md
├── database-connection-failures.md
├── kafka-failures.md
└── maven-build-failures.md
Each note covers typical failure patterns, causes, diagnostic steps and recommended solutions. Ingestion turns the Markdown files into searchable vectors.
Markdown Documents
|
v
Document Loader
|
v
Chunking
|
v
Embedding Model
|
v
PostgreSQL + PGVector
At query time, the failure is rewritten as a semantic query and matched against those vectors.
Build Failure
|
v
Create Semantic Query
|
v
Embedding Model
|
v
PGVector Similarity Search
|
v
Relevant Knowledge
Whatever comes back can serve as context for the model.
A deliberately local stack
The first version was meant to stay local and modest. It runs on PostgreSQL with the PGVector extension, Ollama and LangChain4j. PostgreSQL lives in a Docker container. Embeddings come from a model that Ollama serves locally.
nomic-embed-text
Each embedding is a long numeric vector.
768-dimensional embeddings
Because the knowledge turns into vectors, it can be searched by semantic similarity rather than keywords.
First proof that retrieval works
An early test used a plain-language query describing the bug.
PaymentValidator is null when processing a payment
The query was embedded and sent to PGVector. Back came the null-pointer troubleshooting note, scoring a similarity of roughly this value.
0.8006
The returned text explained that a NullPointerException appears when code uses an object reference holding null. Also retrieved was the Spring injection example, a service class with a field that never gets wired up.
@Service
public class PaymentService {
private PaymentValidator validator; public void process(Payment payment) {
validator.validate(payment);
}
}
Alongside it came the advice to switch to constructor injection. The milestone mattered because the system was no longer merely storing documents. It was finding relevant knowledge by the meaning of a query.
Chunking instead of whole documents
At first, each Markdown file became one single embedding. That's fine for a proof of concept and wrong for production. The intended flow splits big documents into smaller chunks.
Large Document
|
v
Chunks
|
+----> Embedding
|
+----> Embedding
|
+----> Embedding
|
+----> Embedding
|
v
PGVector
With chunks, retrieval returns the most relevant slices of knowledge instead of an entire file. Recursive chunking with overlap is being tried, so important context doesn't get cut off at chunk edges.
The stack, piece by piece
- Java 21 with Spring Boot forms an ordinary backend exposing REST endpoints.
- LangChain4j links the Java application to the embedding model, the LLM and the vector store.
- Llama 3.2 through Ollama runs locally, so the project doesn't depend entirely on a hosted API.
- A nomic-embed-text model, also via Ollama, turns documents and queries into vectors.
- PostgreSQL with PGVector stores embeddings and answers similarity searches.
- Docker hosts PostgreSQL and PGVector during local development.
What already works
Detection covers compilation failures, Maven dependency problems, automated test failures and runtime failures, with dedicated analysis for null-pointer and illegal-argument exceptions. Extraction handles components, source locations, stack traces and ANSI-colored log cleanup.
On the reasoning side there are structured LLM responses, root-cause analysis, remediation advice and confidence-based recommendations.
The knowledge layer includes the Markdown knowledge base, ingestion, chunking, local embedding generation, PostgreSQL and PGVector storage, similarity search and retrieval of relevant troubleshooting material.
Wiring retrieval into the model
The next milestone connects retrieved knowledge to the LLM. The target flow starts with a failure and ends in a recommendation.
Build Failure
|
v
Rule-Based Analysis
|
v
Structured Data
|
v
RAG Retrieval
|
v
Relevant Knowledge
|
v
Llama 3.2
|
v
Context-Aware Diagnosis
Rather than asking the model an open question about the build,
"What is wrong with this build?"
the analyzer will supply the failure, its error type and the retrieved knowledge as a package.
Build failure
+
Error type
+
Component
+
Source location
+
Stack trace
+
Retrieved engineering knowledge
The model is then asked to reason over that context.
Toward an engineering agent
RAG is one stage of a longer design. Today the path is short.
Build Log
↓
Diagnosis
The eventual shape is far richer.
Build Failure
|
v
Diagnosis
|
v
Repository Analysis
|
v
Find Relevant Code
|
v
Generate Fix
|
v
Create Git Branch
|
v
Create Pull Request
|
v
Run CI
|
+------ Failed ------+
| |
| v
| Investigate
| |
+--------------------+
|
+------ Passed ------> Successful Fix
That introduces an AI engineering agent, which could use tools for inspecting a repository, running Git operations, searching the knowledge base, talking to Jenkins, executing builds and tests, and opening pull requests. The key difference is that such an agent wouldn't just emit an answer. It would investigate, act, observe the outcome and try again.
Lessons so far
Useful AI engineering takes more than calling an LLM. A working system has several layers.
Software Engineering
+
Data
+
Retrieval
+
Models
+
Tools
+
Validation
The deterministic layer matters. So does retrieval, and so does the quality of the knowledge base. Most important is the ability to check an AI-generated recommendation. That's the reason for building step by step instead of leaping to an autonomous agent.
For a Java backend developer, a project like this forces the engineering questions to the surface: not how to call an LLM API, but how to fit AI into a real software workflow. It covers LLM integration, embeddings, vector databases, RAG, semantic search, AI-assisted engineering, agentic workflows, automated remediation and CI/CD integration. Development continues, but the progression is already clear.
Build Logs
↓
Deterministic Analysis
↓
LLM Reasoning
↓
Embeddings
↓
PGVector
↓
Semantic Retrieval
↓
RAG
↓
Repository Intelligence
↓
AI Agent
↓
Automated Remediation
That ladder from logs to diagnosis, and eventually to action, sets the direction for what comes next.