Graphify, OKF and RAG: Which Context Layer Does Your Codebase Need
Benchmarks show Graphify ties vector RAG on accuracy. Here is how Graphify, Google's OKF and vector RAG fit together, with failure triggers and a decision matrix.
This article is published in English.
Graphify's own benchmark file contains a result that the hype around the project tends to skip. On head-to-head question-answering accuracy it ties with dense vector RAG, and on recall it trails slightly. Yet it's been called a "RAG killer" across developer blogs, and enterprise teams are attaching it, along with Google's Open Knowledge Format (OKF), to their AI coding agents in the very same month. Why?
Because the viral framing is backwards. Graphify and OKF don't compete to replace RAG. Both are specialized layers, one structural and one narrative, that sit next to a vector index, and each one is justified only by a particular failure mode in your system. Judge them as RAG replacements and you'll burn engineering time maintaining knowledge graphs and curated markdown bundles for problems you don't have.
What follows covers what each layer does, why Graphify's real edge is ingest cost and not superior retrieval, and a decision matrix for a 2026 codebase.
What the benchmark file actually says
Open BENCHMARKS.md in the Graphify repository and the numbers launch posts leave out are right there.
Take LongMemEval-S first, with 50 samples. Graphify's graph-expand mode gets 76% QA accuracy and 0.844 recall@10. A dense vector RAG baseline with the same model (Kimi K2.6), the same BGE-m3 embedder and equal token budgets also gets 76%, with 0.848 recall@10. So knowledge graphs don't win here: accuracy is tied and recall is a touch lower. Worse for the hype, the vendor itself reported the dataset and validated it blind (90.6% grading agreement, Cohen's kappa 0.81), so no hostile outsider picked a bad run.
On LOCOMO (n=300), the picture is similar, though the metric that matters moves to ingestion cost.
Graphify scores 45.3% accuracy and costs about $1.40 to ingest the dataset. Supermemory scores higher at 49.7% but costs $15.67 to ingest, roughly 11 times as much. So Graphify isn't winning the accuracy race here either. It wins on cost and efficiency.
Across both public benchmarks the pattern holds. Graphify doesn't out-retrieve a well-tuned vector RAG pipeline. It matches vector RAG on accuracy while costing far less to build than other memory systems. Two properties explain its advantage:
- Cheap ingest, because AST parsing needs no LLM API calls at all.
- Structural extraction with no hallucination risk, since deterministic tree-sitter parsing pulls exact code relationships (calls, imports, class inheritance) with no model involved in that pass.
For anyone who's been reading that Graphify replaces vector databases, this reframes the question. The decision isn't which tool wins. It's which layer owns which context, and which failure mode in your agent stack is still unaddressed.
Three layers rather than three rivals
Sound architecture choices need technical reality separated from marketing language. Modern agent setups lean on three distinct context layers.
┌─────────────────────────────────────────────────────────────────┐
│ AI CODING AGENT CONTEXT │
└────────────────────────────────┬────────────────────────────────┘
│
┌─────────────────────────────┼─────────────────────────────┐
▼ ▼ ▼
┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐
│ VECTOR RAG │ │ GRAPHIFY │ │ GOOGLE OKF │
│ (Unstructured) │ │ (Structure) │ │ (Narrative) │
├───────────────────┤ ├───────────────────┤ ├───────────────────┤
│ • Docs & Tickets │ │ • AST Call Graphs │ │ • Architecture │
│ • Slack Threads │ │ • Class Hierarchies│ │ • Data Schemas │
│ • Long Tail Text │ │ • Imports/Exports │ │ • Playbooks & Run │
└───────────────────┘ └───────────────────┘ └───────────────────┘
Vector RAG
Vector RAG cuts unstructured text into fixed-size token blocks (512 tokens, say), embeds them in a vector space, and at query time returns the top-k nearest neighbors. It's quick, cheap and a good match for independent, flat documentation.
Its basic flaw is structural: it flattens hierarchy. A 2026 analysis from Prism Labs puts it this way: when cause and effect don't sit in the same 512-token chunk, the system has no idea they're linked. Better embedding models can't repair that, because it's inherent in chunking text.
Graphify
Graphify is a local, deterministic engine. It runs tree-sitter AST parsing over more than 36 programming languages, including Python, TypeScript, Go, Rust, Java, C/C++, Swift, SQL and Terraform/HCL, pulls out the code elements and assembles a structural graph with typed edges like calls, imports, inherits and mixes_in.
Each edge carries a label. EXTRACTED means it was read straight from the AST with full confidence, and INFERRED means a heuristic produced it with lower confidence. On top of that, Graphify applies Leiden community detection to separate natural subsystems and flags "god nodes", highly central files that touch many modules.
Running /graphify . yields three local artifacts:
graph.htmlis an interactive visual of the graph.GRAPH_REPORT.mdis a written overview of the codebase's subsystems.graph.jsonis the machine-readable, queryable index.
Open Knowledge Format
Google Cloud announced OKF in June 2026. It isn't a retrieval algorithm or a database engine. It's a vendor-neutral markdown metadata specification. A bundle is a directory of plain UTF-8 Markdown files with minimal YAML frontmatter, where just one field, type, is mandatory. A header looks like this:
---
type: Architecture Concept
title: Payment Gateway Routing
description: Canonical sequence for multi-tenant payment provider dispatch.
tags: [billing, architecture]
---
There's no schema registry and no heavy runtime. Bundles are meant to live in Git and be read with ordinary shell tools like cat. What OKF holds is curated institutional memory, written by people or drafted by agents: the stable narrative of why systems are built as they are, and not raw code symbols or uncurated chat logs.
How the layers divide the work
Combined, they split the work: Graphify extracts what the code is, OKF captures what engineers know, and vector RAG handles the unstructured remainder.
From a tweet to a 63K-star repo in 90 days
The rush to adopt these tools makes more sense against the timeline, because the whole ecosystem appeared inside a single 90-day window.
2026-04-01: Karpathy posts "LLM Wiki / Knowledge Compiler" concept on X
│
2026-04-03: Safi Shamsi launches Graphify on GitHub (YC S26)
│
2026-06-01: Graphify reaches 58.3K GitHub stars & 1.2M PyPI downloads
│
2026-06-12: Google Cloud releases Open Knowledge Format (OKF) spec
│
2026-07-01: Graphify ↔ OKF hybrid integration toolkit ships (71.5x token claim)
│
2026-07-05: Graphify hits ~78K stars; Level Up Coding scrutinizes "70x token" claims
The same sequence in prose:
- April 1, 2026: Andrej Karpathy shares a thought experiment on X: put research papers, codebase notes, screenshots and architecture documents in a single folder, then query them through an AI that understands how they relate, with LLMs playing "knowledge compilers".
- April 3, 2026: Safi Shamsi publishes Graphify on GitHub, turning that idea into a project within 48 hours. It joins Y Combinator's S26 batch.
- June 1, 2026: the project crosses 58,300 stars on GitHub and 1.2 million downloads from PyPI.
- June 12, 2026: Google Cloud unveils OKF together with its open spec, sample bundles (GA4 e-commerce, Stack Overflow and Bitcoin) and an agent that extracts BigQuery metadata.
- June 15, 2026: Knowledge Catalog from Google Cloud now ingests OKF bundles natively and hands them to enterprise AI agents.
- July 1, 2026: a toolkit connecting Graphify and OKF in both directions arrives. Graphify code graphs get exported as OKF markdown catalogs through progressive disclosure, with claimed query-token savings of up to 71.5x compared with reading raw files.
- July 5, 2026: Graphify nears 78,000 stars, and technical publications, Level Up Coding among them, begin questioning the headline efficiency claims.
Andrus, writing during that period of scrutiny, noted that a result near 70x savings hinges on one property of a project which almost no post states openly. The property is corpus topology. The test behind the 70x figure used one specific 52-file set (Karpathy's public repos, 5 research papers, 4 diagrams), and on enterprise codebases with flat package structures or huge monorepos, token savings vary widely.
Which failures justify adding either tool
Hype isn't an architectural requirement, and neither is vague unhappiness with the current RAG setup. New infrastructure needs a concrete, observed failure mode behind it.
A reality check first
Surveys across the industry put the share of enterprise RAG deployments that underdelivered or failed outright within a year at 72%. Most of those failures come from poor chunking, outdated vector indices or ill-suited embedding models, not from lacking a graph database. A knowledge graph bolted onto a broken retrieval pipeline gives you the same broken pipeline plus more to run.
┌─────────────────────────────────────────┐
│ Does your AI coding agent underperform? │
└────────────────────┬────────────────────┘
│
┌──────────────────┴──────────────────┐
▼ ▼
[ Sifting multi-file code ] [ Loss of tribal context ]
[ dependencies & calls ] [ across team silos ]
│ │
▼ ▼
Multi-Hop Reasoning Failure Knowledge Sprawl Failure
│ │
▼ ▼
TRIGGER: Add Graphify TRIGGER: Add Google OKF
(AST Structural Graph) (Curated Narrative Spec)
Trigger one: multi-hop reasoning failures (add Graphify)
This one shows up when an answer depends on connecting facts across several files that never share a text chunk.
A typical symptom: the agent can't answer questions such as "what breaks in the API layer if the schema in db/migrations/04_users.sql changes?"
Because vector RAG embeds each block independently, it knows nothing about foreign key references, interface implementations or cross-file calls. Better prompts or larger context windows won't help, as those relationships never made it into vector space.
Trigger two: scattered tribal knowledge (add OKF)
This one appears when important institutional context is spread over Slack channels, outdated wiki pages and unwritten developer memory.
The symptom here is an agent that writes syntactically valid code which breaks internal architectural guidelines, calls deprecated internal APIs or ignores domain-specific business rules.
Graphify can't help, because AST parsers read only source code and can't extract business intent. OKF fills the gap with a lightweight, Git-versioned markdown structure where engineers write down stable system conventions that agents can parse reliably.
Research on legal-domain RAG found that standard retrieval tools hallucinate on 17% to 33% of complex queries, since retrieval alone doesn't model logic that spans documents. Atlan's guidance sums up the sensible order: RAG first, and a graph once depth becomes the bottleneck.
What each tool costs when it breaks
Every added layer brings its own failure modes. Both projects are young, so their operational risks deserve attention before production use.
Where Graphify falls short
- Stale snapshots. The graph index is built once and static. It doesn't refresh as developers edit files, so without a Git pre-commit hook or CI/CD step it drifts from the code. A stale structural graph is worse than none, because the agent gets confident wrong answers.
- Noisy communities in legacy monoliths. In large Java or C++ applications, Leiden detection often groups unrelated modules together just because they all use shared infrastructure such as logging or telemetry singletons.
- Hallucination in semantic enrichment. The core code pass is deterministic, but the optional semantic pass over documentation, PDFs and images still calls an LLM. Relationships extracted from non-code artifacts can therefore be hallucinated.
- Unverified claims. According to software architect Saurabh Sharma, nobody outside the project has reproduced a comparison of Graphify graph queries against raw files or a RAG baseline on a shared test set. Vendor numbers are best read as a rough direction.
Where OKF falls short
- No enforcement or validation. The format deliberately leaves out schema registries and strict rules, so a bundle's quality rests entirely on whoever or whatever writes the files.
- Quiet decay. The core contract tells agents to tolerate unknown metadata types and broken links. That speeds adoption, but it lets bundles slide into unlinked, outdated files without any build error.
- Upkeep burden. Markdown docs need ongoing discipline. If engineering workflows don't require OKF updates alongside code changes, the narrative layer goes stale fast.
The combined pipeline in practice
For complex enterprise codebases, the best 2026 setup doesn't pick one tool. It chains them into a multi-stage context pipeline.
┌──────────────────────────────────────────────────────────────────┐
│ USER / AGENT QUERY: "What breaks if I modify UserService.py?" │
└─────────────────────────────────┬────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────┐
│ STAGE 1: OKF Progressive Disclosure Catalog │
│ Read high-level module narrative & owner specs (~2K tokens) │
└─────────────────────────────────┬────────────────────────────────┘
│ (Targeted node identified)
▼
┌──────────────────────────────────────────────────────────────────┐
│ STAGE 2: Graphify Structural Graph Query │
│ Traverses AST edges: calls, imports, inherits (Zero-LLM cost) │
└─────────────────────────────────┬────────────────────────────────┘
│ (Unstructured edge cases)
▼
┌──────────────────────────────────────────────────────────────────┐
│ STAGE 3: Vector RAG / Sparse Search │
│ Search raw Slack incident logs, GitHub issues, & PR comments │
└──────────────────────────────────────────────────────────────────┘
Step one: generate Graphify's structural artifacts
Install Graphify in an isolated environment, then run the analyzer against the workspace.
# Install Graphify via uv
uv tool install graphifyy
# Register the skill with your local agent environment (Claude Code, Cursor, Codex, Gemini CLI)
graphify install
# Execute AST graph compilation inside your repository root
/graphify .
# Generated Local Artifacts:
# graph.html --> Interactive visual network representation
# GRAPH_REPORT.md --> Natural language summary of communities & god nodes
# graph.json --> Queryable AST relational graph database
When that's finished, the local structural index can be queried directly to inspect relationship paths.
# Query dependency paths between components
graphify path "UserService" "DatabasePool"
# Trace architectural blast radius before refactoring
graphify query "What components depend on AuthTokenValidator?"
Step two: write curated concepts as OKF
Export what the structural pass found into an OKF-compliant concept bundle under .well-known/okf/ or docs/knowledge/. A minimal concept file uses ordinary Markdown links, and those links map straight onto knowledge graph edges.
---
type: Database Table Context
title: orders_fact
description: Canonical order transaction table joining billing and fulfillment streams.
resource: bigquery://production-data.commerce.orders_fact
tags: [finance, core-model]
timestamp: 2026-06-20T14:00:00Z
---
# Business Domain Summary
The `orders_fact` table serves as the primary source of truth for revenue calculations.
Modifications to columns must align with the [Customer Profile Spec](./customer_profile.md).
# Schema & Join Constraints
| Column Name | Data Type | Structural Constraint |
|---|---|---|
| `order_id` | STRING | Primary Key; joins with [billing_events](./billing_events.md) |
| `customer_id` | STRING | Foreign Key; see [customers](./customers.md) |
| `status` | STRING | Enum: `pending`, `completed`, `refunded` |
# Related Runbooks
- Incident Protocol: [/runbooks/billing-reconciliation.md](/runbooks/billing-reconciliation.md)
Search and context engineer Kopp makes the point that OKF's link structure fits GraphRAG neatly, since a link from orders_fact.md to customers.md already is an edge in a knowledge graph.
Evidence from ERPNext
The combined approach was tested on ERPNext, an open-source enterprise codebase with over a million lines of Python and JavaScript.
The evaluations used Claude Opus 4.8, capped at 14 turns per task:
- A baseline coding agent using vector RAG and grep reached 70.8% key-fact coverage.
- The Graphify and OKF hybrid pipeline reached 82.0% key-fact coverage, averaging around 140,000 tokens per query session.
An 11.2-point gain on a codebase of that size means noticeably fewer agent execution errors. It says more about production value than synthetic token-reduction numbers do.
Choosing by codebase type
Find your codebase environment and its main failure mode in the matrix to pick an architecture.
The short version
Graphify is strongest at deterministic structural parsing with no LLM API cost. OKF supplies a portable, Git-friendly standard for context written by both people and agents. Vector RAG is still the most effective option for searching unstructured text. Stacking layers without a clear failure mode adds complexity and does nothing for agent performance.
Define the problem first
Before Graphify or OKF goes into a development pipeline, decide which of three problems is real: stale code context, multi-hop dependency failures, or fragmented institutional knowledge.
Both tools are useful additions to a modern AI stack, and the ERPNext results show real value on complex codebases. Still, the leap from one tweet to a viral repository is a reason for rigor. Read the data in BENCHMARKS.md directly, test candidate architectures on your own code, and assemble a context stack that suits your engineering needs. Matching your setup to monolith, microservices or monorepo conditions in the matrix is a good place to begin.