RAG-Anything on 65 Wine Books: Where Graph Retrieval Wins and Fails

The surest way to judge a framework is to build something real with it, hit its limits and measure what happens, instead of trusting benchmarks or a README. Here RAG-Anything gets that treatment. It's an open-source HKU project that claims to produce a knowledge graph automatically from multimodal documents, and the test material is a personal library of wine books gathered by someone who sat the WSET course and exams and just likes the subject.

Why wine makes a good stress test

Wine is a naturally relational domain. Tannins shape how well a wine ages, acidity interacts with tannins, and yeast drives the fermentation that produces flavor compounds. Grapes link to regions, regions to climates and climates to styles. If structured entity-relation extraction can beat flat vector search anywhere, it should do so here. The question under test: does RAG-Anything build a useful knowledge graph from real PDF books, and does it answer wine questions better than naive vector search?

Under the hood, RAG-Anything wraps LightRAG. On top of it come multimodal parsing (PDF, DOCX, images), automatic extraction of entities and relations into a graph, and five query modes that balance graph traversal against vector retrieval. Parsers include MinerU, Docling and PaddleOCR, several LLM backends work, and tables, images and equations get handled at ingestion time. With over 15,000 GitHub stars it has real attention, though stars are hardly the first metric to check. Ahead: measured results, adaptable code and a picture of when graph RAG pays off and when it doesn't.

Architecture

There are two phases, ingestion and querying. Ingestion moves a PDF through this pipeline, starting with a document parser, either MinerU (VLM-based) or the lighter, cheaper Docling:

PDF Document
    |
    v
[Document Parser]  ─── MinerU (VLM-based) or Docling (lighter/cheaper)
    |
    v
[Content Extraction]  ─── text, tables, images, equations
    |
    v
[Text Chunking]  ─── split into manageable pieces
    |
    v
[LLM Entity/Relation Extraction]  ─── LLM extracts entities + relationships
    |
    ├──> [Knowledge Graph]  ─── entities as nodes, relations as edges (GraphML)
    └──> [Vector Embeddings]  ─── chunks embedded for similarity search (JSON)

An LLM reads every chunk and picks out entities such as "Cabernet Sauvignon," "malolactic fermentation" and "Bordeaux," plus the links between them: Cabernet Sauvignon grows in Bordeaux, and malolactic fermentation softens acidity. The results are stored as nodes and edges in a persistent GraphML graph.

Querying offers five modes, each with its own retrieval strategy. The table lists what each one searches, what it suits and where it's weak:

+--------+------------------------------+-------------------------------+---------------------------------+
| Mode   | What It Searches             | Best For                      | Weakness                        |
+--------+------------------------------+-------------------------------+---------------------------------+
| naive  | Vector similarity only       | Factoid questions, robustness | Misses relational structure     |
| local  | Graph neighborhood traversal | Entity-specific deep dives    | Blind to entities not extracted |
| global | Community-level summaries    | Broad thematic questions      | Less specific, slower           |
| hybrid | local + global               | Balanced depth and breadth    | No vector fallback              |
| mix    | Graph + vector together      | General-purpose (recommended) | Slowest mode                    |
+--------+------------------------------+-------------------------------+---------------------------------+

For the models, GPT-4o-mini handles text generation, which keeps the hundreds of LLM calls during entity extraction affordable. text-embedding-3-small produces embeddings, and image processing is turned off to hold costs down:

from openai import AsyncOpenAI
from lightrag.utils import EmbeddingFunc
from raganything import RAGAnythingConfig

aclient = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))

# Text LLM - handles entity extraction and answer synthesis
async def llm_model_func(prompt, system_prompt=None, history_messages=None, **kwargs):
    messages = []
    if system_prompt:
        messages.append({"role": "system", "content": system_prompt})
    if history_messages:
        messages.extend(history_messages)
    messages.append({"role": "user", "content": prompt})
    response = await aclient.chat.completions.create(
        model="gpt-4o-mini", messages=messages, temperature=0.0,
    )
    return response.choices[0].message.content

# Embeddings - 1,536 dimensions, 8K token window
async def _embed_texts(texts, **kwargs):
    response = await aclient.embeddings.create(model="text-embedding-3-small", input=texts)
    return np.array([item.embedding for item in response.data])

embedding_func = EmbeddingFunc(embedding_dim=1536, max_token_size=8192, func=_embed_texts)

# Configuration - Docling parser, tables enabled, images disabled for cost
rag_config = RAGAnythingConfig(
    working_dir="./rag_storage",
    parser="docling",
    enable_image_processing=False,    # skipping images - valid for my use-case
    enable_table_processing=True,
    enable_equation_processing=True,
)

Cost is a real tradeoff. All pages of all books are parsed, chunked and pushed through GPT-4o-mini for extraction, and enabling image handling would stack GPT-4o vision calls on top. For this experiment, text and tables sufficed.

The corpus and the parsers

The collection was 65 PDFs chosen to cover different angles. The WSET books are structured teaching material with tables and technical vocabulary, while others are chatty, built from listicle-style tips or stories about winemaking. That mix tests whether the graph can synthesize across writing styles and complexity levels.

MinerU, the default parser, relies on a vision-language model to read page layout. It's thorough but slow. First use downloads the VLM in about 7 minutes, and every page then takes roughly 15 seconds on CPU, so a 50-page book spent around 20 minutes in parsing before extraction started. The 300-page books crashed it entirely, even when cut into 50-page windows, because OCR timed out.

Docling, a lighter parser with no VLM requirement, handled the same book without a crash, and the whole extraction pipeline finished, so it was used for the remaining books. The practical lesson is that MinerU gives richer layout understanding, such as spotting figures and telling headers from body text more accurately, while Docling is the realistic choice for CPU-only setups. A rerun on a local GPU is planned.

Ingestion code is short, with one call per document:

from raganything import RAGAnything

rag = RAGAnything(
    config=rag_config,
    llm_model_func=llm_model_func,
    embedding_func=embedding_func,
)

for pdf_path in sorted(Path("./data").glob("*.pdf")):
    file_start = time.time()
    await rag.process_document_complete(
        file_path=str(pdf_path),
        output_dir="./output",
    )
    print(f"Completed {pdf_path.name} in {time.time() - file_start:.1f}s")

await rag.finalize_storages()

The framework handles parsing, chunking, entity extraction, graph building and vector embedding internally. The API is well designed, almost to the point of oversimplifying.

All the books were extracted in 6 hours at an average of 30 pages a minute. Parsing isn't the bottleneck. The hundreds of LLM calls for entity extraction are, since each chunk passes through GPT-4o-mini for entities and relationships. Even so, the LLM bill without images was small, around $5 for the whole experiment including evaluation runs. The table after ingestion summarizes the graph's size:

+----------------------------+------------------------------+
| Metric                     | Value                        |
+----------------------------+------------------------------+
| Entities (graph nodes)     | 37,132                       |
| Relations (graph edges)    | 47,650                       |
| Text chunks                | 6,247                        |
| Storage on disk            | ~185 MB (all JSON + GraphML) |
| Indexed documents          | 65                           |
| Load time at query startup | ~10 seconds                  |
+----------------------------+------------------------------+

No external database is involved. A GraphML file holds the graph, with JSON files for the vector stores and key-value data. The complete knowledge base fits into a Docker image with no supporting infrastructure.

Graph versus vector

The central experiment sent 6 queries through two modes, local (graph traversal alone) and naive (vector similarity alone), then compared answers. Queries were picked to exercise relational, comparative, causal and cross-document reasoning.

async def compare_modes(rag, query):
    """Compare different retrieval modes on the same query."""
    for mode in ["local", "naive"]:
        result = await rag.aquery(query, mode=mode)
        print(f"[{mode}] {len(result)} chars, {elapsed:.1f}s")

Results for each query, with timings, are laid out below.

+---------------------------------------------------+----------------------+---------------------+-------------------------------+
| Query                                             | Graph (local)        | Vector (naive)      | Winner                        |
+---------------------------------------------------+----------------------+---------------------+-------------------------------+
| How does soil type influence wine character?      | 32.5s / 3,464 chars  | 28.2s / 2,531 chars | Tie                           |
| Relationship between tannins, acidity, and aging? | 24.8s / 2,266 chars  | 25.8s / 2,289 chars | Graph (structure)             |
| Compare red vs. white winemaking                  | 32.9s / 3,119 chars  | 42.0s / 3,423 chars | Graph (speed + structure).    |
| What role does yeast play in fermentation?        | 27.8s / 2,587 chars  | 26.8s / 2,403 chars | Tie                           |
| How do fortified wines differ from table wines?   | 28.5s / 2,635 chars  | 23.1s / 2,597 chars | Tie                           |
| Sparkling wine production methods?                | 10.7s / 48 chars     | 48.4s / 2,900 chars | Vector (graph fails)          |
+---------------------------------------------------+----------------------+---------------------+-------------------------------+

Tannins, acidity and aging. A classic relational question. The graph has explicit edges from tannins to aging potential, from acidity to freshness, and from both to quality. Walking those edges yields a structured account of how tannins and acidity interact, which otherwise means combining chunks that may lie far apart in the source text.

Red versus white winemaking. The graph expresses the contrast through entity links: red and white winemaking each connect to shared process entities (fermentation, pressing, maturation) over different relationship edges. It answered in 32.9 seconds where vector search took 42.0, with a more organized result.

Fortified versus table wines. This needs cross-document synthesis across several books, and the graph connected entities across document boundaries smoothly.

On simple factoid questions, like how soil shapes wine character or what yeast does in fermentation, the two modes gave comparable answers. Chunks contain that information directly, so vector similarity retrieves the passages as effectively as traversal.

The failure that matters most

On a question about sparkling wine production methods, graph mode answered that it didn't have enough information. That reply was 48 characters and took 10.7 seconds. Vector mode returned a detailed answer of 2,900 characters covering the Traditional, Charmat, Transfer and Ancestral methods, in 48.4 seconds.

Entity extraction had missed the key sparkling concepts. No nodes existed for "methode champenoise," "tank method" or "riddling," so a graph query for sparkling wine production found nothing to traverse. The graph was blind to the topic. Vector search simply located text chunks containing those terms and passed them to the LLM, and it tolerates extraction failures because it works on raw text instead of a curated entity set.

The basic limitation follows: when extraction misses a concept, graph queries can't see it, so the graph is only as complete as the extraction pipeline. Vector search has no such failure mode. That's why the mix mode exists, and why local alone is a poor choice.

Timing deserves a look too. Graph queries took from 10.7 seconds (the failure, which found nothing quickly) up to 32.9, and vector queries from 23.1 to 48.4. LLM synthesis dominates both, not retrieval, and caching makes repeated queries on similar topics much faster.

Looking at the graph

To inspect the graph itself, an interactive visualization built with NetworkX and pyvis showed the 120 most-connected hub entities:

import networkx as nx
from pyvis.network import Network

G = nx.read_graphml("rag_storage/graph_chunk_entity_relation.graphml")

# Focus on the most connected nodes (hubs)
degree_dict = dict(G.degree())
top_nodes = sorted(degree_dict, key=degree_dict.get, reverse=True)[:120]
subG = G.subgraph(top_nodes).copy()

# Categorize nodes by wine domain keywords
def categorize(name):
    lower = name.lower()
    if any(k in lower for k in ["cabernet", "merlot", "pinot", "riesling", ...]):
        return "grape"       # Red nodes
    if any(k in lower for k in ["bordeaux", "california", "champagne", ...]):
        return "region"      # Blue nodes
    if any(k in lower for k in ["fermentation", "aging", "maceration", ...]):
        return "process"     # Green nodes
    if any(k in lower for k in ["port", "sherry", "sparkling", ...]):
        return "wine_type"   # Orange nodes
    return "general"         # Purple nodes

The picture colors nodes by category (red for grapes, blue for regions, green for processes, orange for wine types, purple for general concepts) and scales node size by connection count.

The 15 most-connected entities show what the graph captured well. "Wine," a general entity, leads with 2,136 connections:

+--------------------+-------------+-----------+
| Entity             | Connections | Category  |
+--------------------+-------------+-----------+
| Wine               | 2,136       | General   |
| Wine Production    | 1,344       | General   |
| Italian Wines      | 768         | General   |
| Bordeaux           | 442         | Region    |
| Cabernet Sauvignon | 420         | Grape     |
| Riesling           | 412         | Grape     |
| Champagne          | 348         | Region    |
| California         | 321         | Region    |
| Grapes             | 287         | Grape     |
| Port               | 243         | Wine Type |
+--------------------+-------------+-----------+

Visible patterns include a dense cluster of fermentation processes, grape varieties linking outward to regions, and a sub-cluster of fortified wine entities (Port, Sherry, Madeira) with their own production nodes. Sparse areas are just as telling. Sparkling wine concepts are underrepresented, which explains the failed query. The collection also holds few books on sparkling wine, so the gap may come partly from the data.

A small demo front end

For sharing and non-technical demos, the query interface can be wrapped in Gradio:

import gradio as gr


def query_wine(question, mode):
    t0 = time.time()
    result = _run_async(_query(question, mode))
    elapsed = time.time() - t0
    return result, f"**Mode:** {mode} | **Time:** {elapsed:.1f}s"

with gr.Blocks(title="Wine Knowledge RAG") as demo:
    question = gr.Textbox(label="Ask a wine question", lines=2)
    mode = gr.Radio(
        choices=["mix", "local", "global", "hybrid", "naive"],
        value="mix", label="Retrieval Mode",
    )
    submit_btn = gr.Button("Ask", variant="primary")
    stats = gr.Markdown("")
    answer = gr.Markdown(label="Answer")
    submit_btn.click(fn=query_wine, inputs=[question, mode], outputs=[answer, stats])

The app loads the prebuilt graph in about 20 seconds and needs no GPU for queries, since all computation goes through API calls to OpenAI.

What the experiment shows

Graph RAG clearly improves answers for three kinds of query:

• Relational questions, asking how one thing influences another: edges between entities mean the answer doesn't require both to share a text chunk. The link between tannins, acidity and aging was a clear win.
• Comparisons between two things: contrasts organize themselves. Red and white winemaking reused process nodes through different relationship edges, which gave tidy side-by-side answers.
• Synthesis over several books: vector search can manage it as well, yet the graph spells out the connections and doesn't depend on embeddings landing near each other.

Three weaknesses stand out:

• Entity extraction is the weak link. Sparkling wine methods were in the raw text yet never became graph entities, a limit of any extraction-based approach.
• Factoid queries don't need a graph. A question like what temperature to store wine at is fine with vector search, and the graph adds overhead with nothing gained.
• Cost and latency are real. Ingestion needs hundreds of LLM calls, and query latency of 25 to 30 seconds is dominated by LLM synthesis.

A decision matrix maps query types to the three options, vector (naive), graph (local) and mix, with mix recommended:

+----------------------------+----------------+---------------+-------------------+
| Query Type                 | Vector (naive) | Graph (local) | Mix (recommended) |
+----------------------------+----------------+---------------+-------------------+
| Factoid lookup             | Good           | Good          | Good              |
| Relational ("X affects Y") | OK             | Best          | Best              |
| Comparative ("A vs B")     | OK             | Best          | Best              |
| Cross-document synthesis   | OK             | Good          | Best              |
| Topic with extraction gaps | Best           | Fails         | Good              |
+----------------------------+----------------+---------------+-------------------+

Four practical takeaways:

1. Use mix mode every time. Graph-only retrieval shouldn't be relied on, and the missing sparkling coverage shows why.
2. Choose Docling over MinerU for CPU-only deployments, since MinerU's VLM is thorough but fragile.
3. Budget time for ingestion, especially for larger corpora.
4. Inspect the graph. Check that the domain's key concepts actually show up as entities, and evaluate structural graph metrics before relying on it.

Verdict

RAG-Anything keeps its main promise. It pulls structured knowledge from PDFs into a graph that clearly improves relational and comparative queries: traversal gave the better-organized tannin-acidity-aging answer, the red-versus-white comparison came back faster and tidier, and synthesis across books from different publishers worked. It's promising but young, with parser reliability and extraction coverage the areas to watch. Anyone weighing it for production should repeat this experiment on their own data, since a README won't show where it breaks.