GraphRAG vs Vector RAG: Why Vector Search Fails at Scale

Retrieval-augmented generation lets a language model work with data it was never trained on. The classic version splits documents into pieces, stores their embeddings in a vector database, and at question time pulls back the pieces closest to the query and drops them into the prompt. For simple question answering over a modest collection, that works well. Pointed at complicated enterprise data, it starts to show cracks.

The trouble isn't in the retrieval step or in the generation step. It comes from how the data is represented once it's turned into vectors.

Where plain vector RAG breaks

Chunks become islands

Once a document is cut into chunks, every chunk floats alone in vector space. It's cut off from the paragraphs around it and from related material in other documents, and whatever structure the original had is gone. At retrieval time, important surrounding context stays behind, and the LLM can mistake a partial fact for the whole truth. Severing those logical links forces the model to judge text without the setting it came from.

No reasoning across documents

Some questions can't be answered by any single passage. Think of "what trends and themes show up across all of our quarterly financial reports?" or "can an employee claim reimbursement for Wi-Fi costs incurred before their joining date?" Microsoft Research calls such questions sensemaking queries. They're broad and need a system to analyze, combine and summarize relationships across a dataset, not extract one fact that already exists.

Vector search is built around local semantic matches: the chunks whose vectors sit nearest the query vector. That's excellent for local questions and fails when the answer is scattered over several chunks and documents. Suppose the answer requires linking fact A in one document to fact B in a second and fact C in a third. Each chunk is fetched independently, based on how similar it is to the query, so a middle fact with a low similarity score gets left out even though the answer depends on it. The chain of reasoning breaks.

Accuracy drops as data grows

Every RAG system that goes from prototype to production grows its corpus, and growth exposes a flaw: retrieval accuracy starts to fall. More documents widen the range of topics the system can cover, but they also dilute precision. When an index moves from hundreds of chunks to thousands, the share of chunks that matter for any one question keeps shrinking. The relevant ones get drowned out by neighbors that look similar semantically yet are useless in context. More documents mean broader coverage and lower retrieval precision at the same time.

GraphRAG as the answer

Microsoft Research introduced GraphRAG in 2024 to address these weaknesses. Rather than disconnected vectors, it keeps the structured relationships hidden in the text by moving to a knowledge graph. A knowledge graph represents real-world facts as nodes, which are entities such as people, places or concepts, linked by edges that spell out how they relate. That foundation lets the system reason about an entire dataset instead of only doing local lookups.

The core idea

GraphRAG converts scattered documents into a connected map of ideas and summarizes the data from broad themes down to details. Because those links are built ahead of time, the LLM gets both a bird's-eye view of the whole corpus and access to fine-grained context. There are four stages:

1. Build the graph. An LLM constructs a knowledge graph in which nodes are the key entities of the corpus and edges are the relationships between them.
2. Find communities. The graph is partitioned into nested clusters, from small local groups up to dataset-wide themes. A community is a set of nodes tightly connected to each other and only loosely tied to the rest of the network.
3. Summarize bottom-up. An LLM writes a summary for each community that covers its entities, facts and relationships. Summaries are produced following the hierarchy, so higher-level communities absorb the summaries of those below them and a structured overview of the entire dataset emerges.
4. Answer with map-reduce. At query time, community summaries are processed in parallel into partial answers (map), which are then merged into one global answer (reduce).

The workflow in detail

Like conventional RAG, GraphRAG has two phases, indexing and querying.

Indexing, done offline

Documents to chunks. The process begins by splitting documents into chunks, typically around 600 tokens, a size chosen to maximize how many entities get picked up.

Chunks to entities and relationships. The LLM reads each chunk in turn and extracts important entities and the relationships among them, along with short descriptions of each. It can also extract claims, which are noteworthy facts about entities such as dates, events and interactions with other entities.

Into a knowledge graph. All extractions are consolidated. Entities become nodes and relationships become edges. To cut redundancy, variants of the same entity are merged and their descriptions condensed into one entry. Repeated mentions of a relationship raise the weight of its edge, signaling a stronger connection, and claims attach to the relevant nodes and edges.

Graph to communities. A hierarchical community detection algorithm, usually Leiden, partitions the network into clusters of closely connected nodes. The result is a multi-level hierarchy with tiny, detailed clusters on the lowest level and sweeping domain-wide groupings on the highest.

Communities to summaries. Working from the smallest, most detailed clusters upward, the LLM writes a summary for each level of the hierarchy. This supports big-picture questions and lets users browse from broad themes to fine detail. To respect context limits, lowest-level summaries are made by ranking raw graph data by importance and filling the prompt with the best-connected entities, relationships and claims until space runs out. Higher-level summaries are built directly from the summaries beneath them, a rollup of rollups.

Querying at runtime

Standard RAG treats every prompt with the same similarity lookup. GraphRAG sends a query down one of two paths depending on intent. Global search handles broad, dataset-wide sensemaking. Local search handles specific, entity-centered questions that need several hops. For a company with a reimbursement policy, a global query might ask about the main compliance risks and approval rules across all reimbursement programs. A local query might ask whether an employee can claim an internet expense incurred before the official joining date.

Global queries. Rather than scanning millions of raw chunks, the system queries the precomputed community summaries using a map-reduce pipeline.

1. Choose a hierarchy level, from broad super-communities to fine local clusters (level 0, 1, 2 and so on), depending on the scope the answer needs.
2. Map: send the question in parallel to every community summary at that level. An LLM evaluates each independently and produces an intermediate community answer with a relevance score from 0 to 100.
3. Filter and rank: discard low-relevance answers and order the rest into prioritized points.
4. Reduce: pass the top-ranked insights to the LLM in one context window, which synthesizes them into a complete, non-repetitive response with citations.

Local queries. GraphRAG doesn't throw vectors away. It changes what they're for. In local search, vectors only locate anchor entities in the graph. Once those entry points are found, vector search stops and graph traversal takes over. As a prerequisite, embeddings for entity descriptions must be generated and stored in a vector database.

This marks a real difference. In traditional RAG, vector similarity is the main way to fetch raw text chunks. In GraphRAG it's a targeted lookup that maps the question to relevant entities, which only act as doorways into the graph. Depending on configuration, local search can also embed and search text units or claims as entry points.

1. Run a vector similarity search against entity description embeddings to find entry points.
2. Map those entities to their source documents and text chunks, walk the graph to collect neighboring nodes and edge descriptions, retrieve covariates such as claims, URLs and dates, and gather the summaries of the local communities the entities belong to, which adds a bit of surrounding context.
3. Have an LLM rank each of these mappings by relevance to the question, dropping the weaker ones.
4. Add the ranked context to the prompt and generate a grounded answer.

Which to choose

Neither approach is better in every case. They suit different questions and data. The decision depends on how complex the dataset is, what queries you need to support and how much you can spend.

Pick vector RAG when you want fast, inexpensive retrieval of isolated facts from standalone documents, where the answer sits within a single paragraph or page. Pick GraphRAG when your sources are dense and interconnected and the questions call for synthesis across documents, trend analysis over a whole dataset or tracing relationships between entities in fields like law, research and policy.

A handy shorthand: vector RAG looks for the needle in a haystack, whereas GraphRAG first charts the whole haystack and only then searches.

Vector RAG changed how LLMs work with outside knowledge, yet scaling to complex enterprise data takes more than semantically similar lookups. By organizing text into a knowledge graph with hierarchical community summaries, GraphRAG connects targeted retrieval with reasoning over a whole dataset.

If you want to read further, the original paper is at https://arxiv.org/abs/2404.16130 and the local search design is documented at https://microsoft.github.io/graphrag/query/local_search/.