Stop Making Your Coding Agent Re-Read the Codebase: The LLM Wiki Pattern

Most coding agents share an irritating habit. Every session starts from zero: they reopen files, rediscover which module depends on which and relearn what each part of the codebase is for. You pay in tokens, and therefore money, for knowledge the agent held five minutes earlier and dropped when the context window closed.

Andrej Karpathy described this problem and sketched a remedy that he called the LLM Wiki pattern. LangChain has released an open-source implementation, openwiki. Below, the pattern and the implementation are examined side by side: what the idea is, why it matters and how the code behind a demo app works. The result is a mental model you can apply to any codebase an agent has to understand.

Agents with amnesia

First, the failure mode in precise terms. The usual answer to "how does an agent understand a codebase?" is RAG. Embed the source files, put them in a vector database and fetch relevant chunks at query time.

RAG is fine for answering questions. For coding agents it falls short in three ways.

Understanding is recomputed every time. Suppose the agent needs the link between agent.py and memory.py. It pulls chunks, the model reasons its way to the answer once more, and the result is thrown away.

Retrieval depends on embeddings. If whoever wrote the code used different vocabulary from the agent's query, the right chunks never come up. Embeddings are approximate, and architecture is exact.

It goes stale without a sound. The vector store has no idea that memory.py was refactored last Tuesday, and it serves old chunks with full confidence.

Karpathy's idea was plain: have the model write the documentation itself, in a format tuned for models to read, and keep it current automatically. That means documentation for agents, not for humans.

Three layers

Every piece of openwiki belongs to one of three layers in Karpathy's design.

┌─────────────────────────────────────────────────┐
│  Layer 3: The Schema                            │
│  AGENTS.md / CLAUDE.md — "the wiki exists here" │
├─────────────────────────────────────────────────┤
│  Layer 2: The Wiki                              │
│  openwiki/*.md — LLM-generated structured docs  │
├─────────────────────────────────────────────────┤
│  Layer 1: Raw Sources                           │
│  Your actual codebase — immutable to the wiki   │
└─────────────────────────────────────────────────┘

Each layer rests on the one beneath it.

Layer 1: the raw sources

The raw sources are simply your codebase. From the wiki's point of view they're immutable: OpenWiki reads them and never changes them.

The demo project keeps its sources in a sample_agent/ directory, a realistic LangChain ReAct agent that most AI engineers will recognise.

# sample_agent/agent.py
from langchain.agents import AgentExecutor, create_react_agent
from langchain_openai import ChatOpenAI
from .memory import build_memory
from .prompts import build_prompt
from .tools import TOOLS
def build_agent(
    model: str = "openai/gpt-4o-mini",
    base_url: str = "https://openrouter.ai/api/v1",
    api_key: str | None = None,
    temperature: float = 0.0,
) -> AgentExecutor:
    llm = ChatOpenAI(model=model, base_url=base_url, api_key=api_key, temperature=temperature)
    prompt = build_prompt()
    memory = build_memory()
    agent = create_react_agent(llm=llm, tools=TOOLS, prompt=prompt)
    return AgentExecutor(
        agent=agent,
        tools=TOOLS,
        memory=memory,
        verbose=True,
        handle_parsing_errors=True,
        max_iterations=10,
    )

Four components are wired together: a model client (ChatOpenAI), a prompt template, conversation memory and the tools. It's textbook ReAct, the sort of code OpenWiki is meant for. The tools are kept deliberately small, a mocked search and a safe calculator:

# sample_agent/tools.py
@tool
def calculator(expression: str) -> str:
    """Evaluate a safe mathematical expression and return the result."""
    allowed_names: dict = {}
    try:
        result = eval(expression, {"__builtins__": {}}, allowed_names)
        return str(result)
    except Exception as exc:
        return f"Error evaluating '{expression}': {exc}"

Look at {"__builtins__": {}}. It strips every Python builtin out of the eval scope so that a user can't pass something like __import__('os').system('rm -rf /') and have it run. A small detail with a real security consequence.

For the pattern, what matters is that OpenWiki will read each of these files, work out how they relate and produce a wiki page explaining it all once, not each time an agent starts up.

Layer 2: the wiki itself

Here sits the central idea. Hand-written docs rot, and RAG over raw code re-derives everything on each query, so the pattern has the model produce structured markdown shaped for the way models take in context.

How OpenWiki produces it

In the demo's Streamlit app, a Generate Wiki button runs this:

# app.py
def run_openwiki(
    args: list[str],
    provider: str,
    api_key: str,
    model: str,
    langsmith: str,
    cf_account_id: str = "",
) -> tuple[str, int]:
    import sys
    env = _build_openwiki_env(provider, api_key, model, langsmith, cf_account_id)
    result = subprocess.run(
        ["openwiki", *args],
        cwd=str(SAMPLE_AGENT_DIR),
        env=env,
        capture_output=True,
        text=True,
        timeout=300,
        shell=sys.platform == "win32",  # npm .cmd files need shell=True on Windows
    )
    return result.stdout + result.stderr, result.returncode

Two details matter.

First, cwd=SAMPLE_AGENT_DIR is deliberate. The openwiki CLI scans whichever directory it runs from. Setting the working directory to sample_agent/ limits the wiki to that codebase and not the whole project. Forget the parameter and it scans from the project root, producing a wiki that covers tests, CI config and node_modules.

Second, shell=True on Windows. Global npm packages install as .cmd wrapper scripts there. Calling subprocess.run(["openwiki", ...]) searches for a binary called openwiki, finds none and raises FileNotFoundError: [WinError 2]. Passing shell=sys.platform == "win32" routes the call through cmd.exe, which resolves .cmd files properly. It's confusing the first time and obvious afterwards.

What "agent-native" formatting means

Human documentation is written to be read from top to bottom. You open README.md, start at the top and build context as you go. Agents don't read that way. They scan headings for relevant sections, pull out facts and move on.

Agent-native docs are shaped around that behaviour:

• Dense headings at every level, so the structure can be scanned without reading prose.
• Explicit cross-references such as See: patterns.md#memory-strategy, which an agent follows mechanically and not by intuition.
• One-paragraph summaries per component: enough to decide whether to read further, not meant to be the full story.
• No padding. Skip the "in this document we will explore" opener and any closing recap.

A generated openwiki/architecture.md for the sample agent might look like this:

# Architecture — sample_agent
## Overview
ReAct agent built on LangChain AgentExecutor. Entry point: agent.py.
Four components: LLM (ChatOpenAI via OpenRouter), tools, memory, prompt.
## Components
- agent.py - AgentExecutor wiring. See: components.md#agent
- tools.py - search (simulated) + calculator (sandboxed eval). See: components.md#tools
- memory.py - ConversationBufferWindowMemory, k=5 turns
- prompts.py - ReAct ChatPromptTemplate with agent_scratchpad placeholder
- evaluator.py - LLM-as-judge chain. See: patterns.md#evaluation
## Data Flow
question → AgentExecutor.invoke() → ReAct loop (max 10 iter)
→ tool calls → final answer → evaluator.evaluate() (optional)

Set that beside a typical human README with three paragraphs of scene-setting before anything concrete shows up. Agents don't need the scene-setting. They need the facts.

Layer 3: the schema, which closes the loop

Generating a wiki is half the job. The other half is making sure the coding agent knows the wiki exists.

That's the role of AGENTS.md and CLAUDE.md, the instruction files that Claude Code, Cursor, GitHub Copilot and most modern coding agents read at the start of a session. OpenWiki injects a short reference into whichever of them your project uses. Before:

# sample_agent/AGENTS.md (before OpenWiki)
This repository contains a LangChain ReAct agent.
## Architecture
- agent.py - AgentExecutor entry point
- tools.py - available tools (search, calculator)
- memory.py - conversation memory strategy
- prompts.py - system prompt and ReAct template
- evaluator.py - LLM-as-judge evaluation chain

And after running openwiki --init:

# sample_agent/AGENTS.md (after OpenWiki injection)
This repository contains a LangChain ReAct agent.
## Architecture
- agent.py - AgentExecutor entry point
...
## OpenWiki Context
When searching for codebase context, check `openwiki/` first.
Key files:
- `openwiki/architecture.md` - overall structure and file responsibilities
- `openwiki/patterns.md` - coding patterns and conventions
- `openwiki/components.md` - detailed component documentation
The wiki is auto-updated daily. Always prefer wiki context over re-reading source files.

The injection is deliberately short. Putting the whole wiki in the instruction file would burn the context window at the start of every session on a large codebase. A pointer is enough. The agent reads it, opens openwiki/architecture.md only when codebase context is needed, and learns what it must without touching a single source file.

The demo's fourth tab shows this before and after, simulating the injection so the mechanism is visible:

# app.py — Tab 4
agents_md_content = AGENTS_MD.read_text(encoding="utf-8")
openwiki_injection = """
## OpenWiki Context
When searching for codebase context, check `openwiki/` first.
...
"""
col_before, col_after = st.columns(2)
with col_before:
    st.code(agents_md_content, language="markdown")
with col_after:
    st.code(agents_md_content + openwiki_injection, language="markdown")

Choosing a model provider

OpenWiki doesn't care which LLM sits behind it. It uses LangChain's ChatOpenAI, which reads OPENAI_BASE_URL for a custom endpoint, so any OpenAI-compatible provider works. The demo supports two:

# app.py
def _build_openwiki_env(
    provider: str,
    api_key: str,
    model: str,
    langsmith: str,
    cf_account_id: str = "",
) -> dict:
    env = {**os.environ, "OPENWIKI_MODEL_ID": model}
if provider == "Cloudflare Workers AI" and cf_account_id:
        cf_base = f"https://api.cloudflare.com/client/v4/accounts/{cf_account_id}/ai/v1"
        env["OPENAI_API_KEY"] = api_key
        env["OPENAI_BASE_URL"] = cf_base      # LangChain reads this automatically
    else:
        env["OPENROUTER_API_KEY"] = api_key   # openwiki's native key name
    return env

OpenRouter is the easy option. A single key opens most major models through one endpoint, and setting OPENWIKI_MODEL_ID=openai/gpt-4o-mini completes the setup.

Cloudflare Workers AI serves models from the edge. ZhipuAI's GLM model (@cf/zai-org/glm-5.2) runs on Cloudflare's own infrastructure, so latency drops when you ship to production and want a single cloud provider. Since Cloudflare exposes an OpenAI-compatible endpoint at /ai/v1, no custom client is needed. Point OPENAI_BASE_URL at it and the LangChain integration already in place does the rest.

Keeping the wiki fresh

Every documentation system eventually fails in the same way: people forget to update it when the code changes.

Karpathy's point was that this isn't a discipline problem but an economics problem. Maintenance cost grows with codebase size, while the person doing it gets no immediate reward. Eventually the docs drift far enough from reality that nobody trusts them, and the cycle starts over.

The answer in OpenWiki is openwiki --update. Rather than rescanning everything, it asks git log which commits landed after the last run, uses git diff to read just the modified files and rewrites only the pages those changes touch. A GitHub Action bundled with the project runs it on a schedule:

# .github/workflows/openwiki-update.yml
name: OpenWiki Update
on:
  schedule:
    - cron: "0 8 * * *"   # 08:00 UTC daily
  workflow_dispatch:
jobs:
  update-wiki:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0   # full git history so openwiki can read diffs
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
      - run: npm install -g openwiki
      - name: Update wiki
        working-directory: sample_agent     # scope to our codebase
        run: openwiki --update --print
        env:
          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
          OPENWIKI_MODEL_ID: openai/gpt-4o-mini
      - uses: peter-evans/create-pull-request@v7
        with:
          commit-message: "docs: update openwiki pages"
          title: "OpenWiki: daily documentation update"
          branch: openwiki/auto-update

Note working-directory: sample_agent, the same scoping choice as in the Python subprocess call. Wherever you deploy this, run --update from the codebase root and not the project root.

Opening a pull request is the right design. Auto-committing to main would let docs change with no review, whereas a PR gives you a diff to inspect and a chance to reject bad model output. The demo's fifth tab simulates the update locally:

# app.py — Tab 5
if st.button("🔄 Run openwiki --update", type="primary"):
    with st.spinner("Checking for changes and updating wiki…"):
        output, returncode = run_openwiki(
            ["--update", "--print"],
            provider,
            api_key if provider == "Cloudflare Workers AI" else openrouter_key,
            model_id,
            langsmith_key,
            cf_account_id,
        )
    st.code(output, language="bash")

An LLM-as-judge evaluator

Alongside the agent comes an LLM-as-judge evaluator, a technique that deserves its own look.

# sample_agent/evaluator.py
def evaluate(
    question: str,
    response: str,
    model: str = "openai/gpt-4o-mini",
    base_url: str = "https://openrouter.ai/api/v1",
    api_key: str | None = None,
) -> dict[str, int | str]:
    llm = ChatOpenAI(model=model, base_url=base_url, api_key=api_key, temperature=0)
    chain = _JUDGE_PROMPT | llm
    result = chain.invoke({"question": question, "response": response})
    try:
        return json.loads(result.content)
    except json.JSONDecodeError as exc:
        raise ValueError(
            f"LLM returned non-JSON response: {result.content!r}"
        ) from exc

temperature=0 is not optional here. The judge should be deterministic, so that the same question and response always get the same score. Random evaluation defeats the purpose of having scores.

json.loads is wrapped in try/except JSONDecodeError because models sometimes put their JSON inside markdown code fences, add a preamble sentence or cut off on long answers. The explicit exception with result.content!r in the message shows exactly what the model returned, which makes debugging quick.

The judge prompt scores accuracy, helpfulness and clarity separately, since they fail independently. An answer may be correct yet fail to address the real question, or point the right way yet confuse the reader. Splitting them gives scores you can act on.

Seeing inside with LangSmith

Set LANGSMITH_API_KEY and every step the openwiki agent takes is traced: which files it read, the prompt it sent, what the model returned and how long each call lasted.

# app.py
def _build_openwiki_env(...) -> dict:
    ...
    if langsmith:
        env["LANGSMITH_API_KEY"] = langsmith
        env["LANGCHAIN_TRACING_V2"] = "true"
        env["LANGCHAIN_PROJECT"] = "openwiki-demo"
    return env

Three environment variables are all it takes. LangChain picks them up automatically with no code change to openwiki or the agent. That's the right observability pattern for LangChain projects: instrument through the environment and not the code, so tracing can be switched on in production without touching the application.

The trace also reveals something useful. The --init call isn't a single LLM call. It's an agent loop of planning calls, file-reading calls and wiki-writing calls, and the trace shows token counts and latency for each. That's how you answer "why does --init take 90 seconds": you open the trace and see three 30-second model calls running one after another.

The full flow

Putting it together:

Developer commits code change
    ↓
GitHub Action runs openwiki --update at 08:00 UTC
    ↓
CLI reads git diff → only changed files
    ↓
LLM regenerates affected wiki pages
    ↓
PR opened with updated openwiki/*.md
    ↓
Developer merges PR
    ↓
Agent starts next session → reads AGENTS.md → sees wiki reference
    ↓
Agent fetches openwiki/architecture.md when it needs codebase context
    ↓
Agent makes informed change without reading source files

The wiki compounds over time. Six months of daily updates leave openwiki/patterns.md holding more than today's structure: it also records how the codebase evolved, including patterns that surfaced in refactors and the reasons behind conventions. A fresh agent or a new hire gets all of that immediately.

The real insight

At bottom the pattern isn't about documentation. It's a question of who bears the cost of understanding a codebase.

Under the RAG model, every agent session pays: re-deriving relationships, relearning conventions, rereading files it has read before. The cost scales with how often sessions happen.

In the wiki model the cost is paid once per code change, by a scheduled job running while nobody waits, at a fraction of a cent per page, and what it produces keeps gaining value.

Models don't tire of keeping wikis current. A deadline doesn't push documentation down their list, and they never leave a "TODO: document later" and forget about it. That imbalance is the entire point.

To try it, install Node.js 18 or later for the openwiki CLI and obtain an API key from either OpenRouter or Cloudflare Workers AI.