RAG or Knowledge Graph? 2,160 Graded Answers and a Model Effect

Why measure at all

Debates about a corporate technical assistant tend to stall at the same spot. One side argues for RAG, the other for a knowledge graph, both bring good arguments, and neither brings numbers. A controlled experiment can supply some: two approaches, three model families, three repetitions and 120 questions with reference answers, producing 2,160 graded responses. Each response was scored by a deterministic metric and by a panel of three judges.

The thesis is less satisfying than anyone would like. There's no absolute winner, the sign of the effect flips depending on the model, and the variable with the largest effect isn't either strategy. It's which model sits behind the system.

For a quick read:

• The graph beat RAG with statistical significance on only one of the three models.
• The judge panel preferred RAG on all three.
• Swapping the model improved quality more than swapping the strategy.
• The dataset is synthetic and the answer key was written by the experimenter, which limits what can be carried over.

The limits come first, because they decide whether the rest is worth reading.

What the study does and doesn't cover

It measures which retrieval strategy yields more correct, more complete and more traceable answers about a microservices ecosystem, with the generating model held constant inside each comparison.

It doesn't measure practical usefulness with real developers, maintenance cost over time, quality as seen by people who know the domain, or behavior on real documentation with its inconsistencies and holes.

It also doesn't measure the ceiling of the graph approach. The graph here is deliberately simple, with one-hop neighborhoods and fuzzy entity matching. A graph supporting multi-hop queries and global aggregation could answer things this one can't, so part of the graph's disadvantage in the tables belongs to the implementation and not to the paradigm.

The problem

A company has moved to microservices. Where one codebase once fit in a single head, there are now dozens of services, contracts, events and databases. The knowledge exists, scattered across READMEs, an internal catalog, an operations board and three people who know it by heart. Everyday questions are mundane and expensive: which application stores personal data, what does service X depend on, does this already exist and can it be reused? The answer today is to ask someone or to read documentation that may be stale, and the real cost isn't search time but a technical decision made on old information.

A conversational assistant would fix access, provided it's specialized in the domain, and the literature offers two routes.

Unstructured retrieval (RAG) slices documentation, indexes it in a vector store, fetches the passages most similar to the question and injects them into the prompt. The term comes from Lewis and colleagues in 2020, and Gao and colleagues survey recent developments. It's cheap to maintain and absorbs change without retraining, but weak when an answer needs reasoning over explicit relationships.

Structured retrieval (a knowledge graph) models the ecosystem as typed nodes and edges: service, team, event, database, dependency. Queries are deterministic and traceable, though modeling and governing the graph is costly. Hogan and colleagues' survey of knowledge graphs is the standard reference.

The two converge in GraphRAG, and that's where this experiment's results point.

Design

The most important decision came before any code: only the knowledge layer may vary. Without that rule, two implementations are being compared and not two strategies. Any difference in prompt or output format contaminates the result in ways that can't be separated afterward, which is why many informal RAG benchmarks end up measuring the skill of whoever wrote the prompt.

Both approaches return exactly the same response contract: question, approach, answer, sources, retrieved context, latency and estimated cost. That keeps the evaluation code, the API and the interface agnostic to the approach.

The test needed a realistic, publishable, reproducible ecosystem, and using an employer's real one was out of the question. So the setup invents CreditXPTO, a personal-credit fintech that doesn't exist:

Being synthetic buys three things. The content is controlled, the correct answer to every question is known, and anyone can rerun the work without access to internal systems. The price is external validity, the main limitation, discussed at the end.

The same knowledge exists in both representations, and equivalence is enforced by validation rules running in continuous integration: a broken reference makes the dataset invalid. Otherwise one side could quietly know more than the other. The 120 questions are labeled by category (factual, explanatory, impact, relational, discovery, ambiguous), by difficulty and by knowledge type, and each has an expected answer and sources.

The RAG side uses 512-token fragments with a 50-token overlap and top-k of 5. Embeddings come from BAAI/bge-small-en-v1.5 running locally, with Qdrant as the vector store. Dense retrieval follows the scheme popularized by Karpukhin and colleagues (DPR), sentence embeddings descend from Sentence-BERT, and the BGE family is described in C-Pack.

The graph side uses NetworkX in memory, built straight from the source files so it always agrees with the data. Entity matching is fuzzy with a 70/100 threshold, and retrieval takes a one-hop neighborhood.

Scoring

Evaluation works on two levels. Level one is heuristic: lexical similarity, presence of key entities and source coverage. It's cheap, deterministic and reproducible, and also shallow, since it punishes paraphrase. That's the known flaw of n-gram overlap metrics and the reason semantic alternatives like BERTScore exist.

Level two is a panel of three LLM judges, one from each model family, scoring correctness, completeness, groundedness and clarity from 0 to 3. The LLM-as-a-judge paradigm was consolidated by Zheng and colleagues with MT-Bench.

One detail matters: the headline metric is cross-family. For every answer, the judge from the generator's own family is left out of the calculation. Models recognize their own text and rate it higher, and Panickssery and colleagues (2024) tie self-preference to that self-recognition. The only way to neutralize it is structurally, never by asking for impartiality in the prompt.

Significance comes from a paired McNemar test, since both strategies answer the same questions. Intervals use bootstrap with 5,000 resamples, and judge agreement is measured with Cohen's kappa.

Results

The first table gives strict accuracy, where only fully correct answers count, with n = 360 per cell:

Two readings matter.

First, the sign flips with the model. RAG wins with two generators, the graph wins with the third, and only that last difference is significant. Running the experiment with a single model, as most benchmarks do, would have produced a confident, wrong conclusion, complete with a confidence interval. Nothing in the result would have hinted at it.

Second, the gap between models exceeds the gap between approaches. Moving from claude-sonnet-4-6 to gpt-5.5 helps more than switching strategy within either model. That demotes "RAG or graph?" to a second-order decision, and it's the study's cross-cutting finding.

Repeating the test with the broad definition, full credit plus partial credit and penalizing only flat-out errors, reverses things. RAG now wins significantly on the first two models, and the graph's advantage on the third stops being significant. The precise reading is that RAG's virtue isn't being right more often but being wrong less badly. When retrieval is partial, it answers incompletely. When entity matching fails, the graph answers wrongly and confidently. For an internal assistant that distinction is everything: a developer can complete an incomplete answer, but a wrong answer with a cited source gets carried into a meeting.

The judge panel results appear next:

The panel prefers RAG on all three models, including the one where the heuristic gave the graph a significant edge. That's an instrument effect and not a contradiction. In one concrete case, a question received a lexical similarity of 0.25 and was classed as incorrect by the heuristic, yet the panel gave it 2.88 out of 3. The answer was right, just phrased in other words. N-gram metrics measure vocabulary, and LLM judges measure perceived correctness, with bias. Neither alone answers the question, and when they disagree the disagreement shows where to look.

The panel deserves an honest caveat: agreement between judges was only moderate, with Cohen's kappa between 0.09 and 0.56. A panel of LLM judges isn't a precision ruler. It's a structured second opinion, and frameworks like RAGAS exist precisely because evaluating retrieval-augmented generation remains an open problem.

On latency, RAG was 12 to 20% faster on all three models. The graph's context is more compact, but the query has an extra step. Cost flipped by model, with the graph cheaper on two generators and pricier on one. The mix of input and output tokens varies enough between providers to change the sign. That's an observation, not a conclusion, since token composition wasn't formally tested.

How each approach fails

This is the most useful part, and it appears in no table.

The graph fails structurally. Asked which database a service uses, it returned the node identifier and not the engine, because the attribute wasn't modeled. Asked how many teams use a given technology, the one-hop neighborhood retrieved two services and it answered confidently about two. And if fuzzy matching picks the wrong entity, the entire context is wrong with nothing signaling it.

RAG fails through incompleteness. It correctly identifies that a service uses Redis but limits the answer to the retrieved service when the question asked for the overall picture.

Both are wrong, in different ways. The defects are complementary, which is exactly why the obvious direction is hybrid, with routing by question type.

Recommendations

• Pick the model first. It's the variable with the largest effect. Treating it as an implementation detail while spending weeks tuning fragment size optimizes a second-order parameter.
• Start with RAG. It has a better floor and cheaper upkeep, and its failure mode is the tolerable one.
• Add structure surgically, for dependencies, ownership and event consumption. Don't try to model the whole ecosystem before delivering anything.
• Don't trust a single metric. Use a deterministic heuristic and a panel from different families. When they disagree, an interesting case has turned up and not a bug.
• Use a paired test. Both strategies answer the same questions, and comparing means without McNemar produces confidence the data doesn't support.

Limitations

• The dataset is synthetic and the answer key is the experimenter's own. No real user validated it. It buys control and reproducibility and costs external validity.
• The graph implementation is deliberately simple, so some of its weakness on aggregation may come from the implementation and not the paradigm.
• Practical usefulness and maintainability weren't measured. They were among the goals and stayed qualitative.
• The heuristic wasn't validated against human judgment. The panel mitigates that without solving it.
• Eleven of the 2,160 responses (0.51%) were provider network errors and not system errors.

The takeaway

The hope was to end with "use a graph" or "use RAG." What came back is that neither answer exists until the question of which model is settled, and that the variable treated as background was the one that weighed most. That's less satisfying and more useful. It also generalizes beyond this case: a benchmark with a hidden variable doesn't err loudly, it errs with a confidence interval. Before choosing an architecture from a number, ask what the experiment held constant, and whether that really should have been constant.