A LangGraph Pipeline That Keeps Codebase Memory for AI Agents Accurate

Anyone who has pointed coding agents at a big repository runs into one question: how can an agent grasp the codebase without reading every file again for each task? What follows is the third try at solving it. Attempts one and two used embedding-based RAG over docstrings. Both decayed within a month, since the docs lagged behind every code change. This third version has coped with a team making thirty or forty commits a day.

It isn't a review of someone else's file format. It's the pipeline itself: a LangGraph state graph that watches a repository, drafts a structured knowledge file for each service, scores its own output for accuracy before any agent reads it and traces every run so the cost is visible. The failures from the first pass are included too.

What it covers: why token-budget trouble in multi-agent coding setups is really a staleness problem, the full LangGraph design (state schema, nodes, conditional edges, DynamoDB checkpointing), a citation pass on Amazon Bedrock with Claude Haiku 4.5, an evaluation gate built on Ragas and DeepEval, observability with self-hosted Langfuse, a fully local variant on Ollama for teams that can't send code to a cloud model, the git hook that ties it together and a reality check on failure modes.

The problem

Whenever a coding agent starts a task on a repo it doesn't know, both of its options are poor. It can read everything, which burns tokens on files unrelated to the job. Or it can use embedding-based retrieval, which returns semantically similar chunks but gives no explicit answer to who calls what or who owns a module.

The alternative that works is a handful of curated, structured files, one per service, stating what it does and what it depends on, generated once and refreshed automatically. An orchestrating agent consults an index, selects the two or three files relevant to the task and then forwards only those to a sub-agent. It's as much about spending tokens wisely as about representing knowledge.

The file format isn't where the difficulty lies. Markdown plus YAML frontmatter is hardly novel. The difficulty is the machinery that stops those files from going stale as the code underneath them moves.

The architecture

                       git push / commit
                               |
                               v
                  +-------------------------+
                  |   scan_diff (node)      |
                  |   git diff --name-only  |
                  |   scope to changed pkgs |
                  +-------------------------+
                               |
                               v
                  +-------------------------+
                  |   draft_concept (node)  |
                  |   Bedrock Claude Haiku  |
                  |   one file per service  |
                  +-------------------------+
                               |
                               v
                  +-------------------------+
                  |   add_citations (node)  |
                  |   link to runbooks/PRs  |
                  +-------------------------+
                               |
                               v
                  +-------------------------+
                  |   eval_gate (node)      |
                  |   Ragas faithfulness    |
                  |   DeepEval assert_test  |
                  +-------------------------+
                          pass |  | fail
                               |  +----> re-draft (max 2 retries) --> human review queue
                               v
                  +-------------------------+
                  |   relink (node)         |
                  |   fix cross-references  |
                  +-------------------------+
                               |
                               v
                  +-------------------------+
                  |   publish (node)        |
                  |   commit bundle to repo |
                  +-------------------------+

        every node emits a Langfuse trace
        every super-step checkpoints to DynamoDB

The graph is deliberately small, with five real nodes and one retry edge. LangGraph invites building something more elaborate than the problem needs. A first attempt with eleven nodes ended up impossible to explain, because nobody could say why a given file existed.

The stack and why each piece is there

+------------------+---------------------------+--------------------------------------+
| Layer            | Tool                      | Why this one                          |
+------------------+---------------------------+--------------------------------------+
| Orchestration    | LangGraph                 | Explicit state, conditional retries,  |
|                  |                           | resumable from checkpoint             |
| Draft model      | AWS Bedrock, Claude       | Cheap enough to run on every commit,  |
|                  | Haiku 4.5                 | fast enough not to block CI            |
| Checkpoint store | DynamoDB                  | Single-digit ms reads, TTL for expiry, |
|                  | (langgraph-checkpoint-aws)| survives a crashed worker mid-run      |
| Eval, faithfulness| Ragas                    | Reference-free scoring of a generated  |
|                  |                           | file against the actual code diff     |
| Eval, CI gate    | DeepEval                  | pytest-native, hard-fails a bad draft  |
|                  |                           | instead of silently publishing it     |
| Observability    | Langfuse (self-hosted)   | Per-node token/cost/latency traces,    |
|                  |                           | MIT licensed, own your own data        |
| Local alternative| Ollama (llama3.1:8b,     | Same pipeline, zero cloud calls, for   |
|                  | nomic-embed-text)         | air-gapped or cost-sensitive teams     |
+------------------+---------------------------+--------------------------------------+

Step 1: the state schema

Everything in LangGraph begins with the shape of the state passed between nodes. Get it wrong and every later node ends up guessing what the one before it produced.

from typing import TypedDict, Literal
from langgraph.graph import StateGraph, END
class ConceptDraft(TypedDict):
    path: str              # e.g. "services/billing-service.md"
    concept_type: str      # "Service", "Runbook", "API Endpoint", ...
    title: str
    body: str              # the drafted markdown body
    citations: list[str]
    faithfulness_score: float
    retry_count: int
class PipelineState(TypedDict):
    repo_root: str
    commit_sha: str
    changed_files: list[str]
    drafts: list[ConceptDraft]
    failed_drafts: list[ConceptDraft]
    status: Literal["scanning", "drafting", "evaluating", "publishing", "done", "needs_review"]

Nothing fancy in there. retry_count is the field that matters, because it prevents a poor draft from cycling endlessly against the eval gate.

Step 2: limit the scan to the diff

Scanning the whole repo on each commit is what makes pipelines like this too costly to run continuously. Limiting the scan to the diff makes them cheap enough for every commit and not just a nightly run.

import subprocess
def scan_diff(state: PipelineState) -> PipelineState:
    result = subprocess.run(
        ["git", "diff", "--name-only", f"{state['commit_sha']}~1", state["commit_sha"]],
        cwd=state["repo_root"], capture_output=True, text=True, check=True
    )
    changed = [f for f in result.stdout.splitlines() if f.endswith((".py", ".go", ".ts"))]
    # map file paths to the service/package that owns them
    packages = sorted({f.split("/")[1] for f in changed if "/" in f})
    state["changed_files"] = packages
    state["status"] = "drafting"
    return state

Mapping files to packages is the one piece that's truly specific to each repo. A monorepo with a tidy services/<name>/ layout needs about three lines. A sprawling legacy repo needs an ownership map kept separately, ideally by parsing a CODEOWNERS file and not by guessing from paths.

Step 3: drafting with Bedrock

This node costs real money, so it deserves deliberate handling. Claude Haiku 4.5 on Bedrock is quick and cheap enough to run per commit, and a bigger model is held back for the human-review escalation path only, never the routine draft pass.

import boto3
import json
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
MODEL_ID = "us.anthropic.claude-haiku-4-5-20251001-v1:0"  # cross-region inference profile
DRAFT_PROMPT = """You are documenting a software service for an AI coding agent's memory.
Given the file contents below, write a concept file with:
- A one-paragraph Responsibilities section
- A Dependencies section listing what this service calls and what calls it
- Nothing else. Do not invent dependencies you cannot see in the code.
Files:
{file_contents}
"""
def draft_concept(state: PipelineState) -> PipelineState:
    drafts = []
    for package in state["changed_files"]:
        contents = read_package_source(state["repo_root"], package)
        response = bedrock.invoke_model(
            modelId=MODEL_ID,
            body=json.dumps({
                "anthropic_version": "bedrock-2023-05-31",
                "max_tokens": 800,
                "messages": [{"role": "user", "content": DRAFT_PROMPT.format(file_contents=contents)}]
            })
        )
        body = json.loads(response["body"].read())
        text = body["content"][0]["text"]
        drafts.append(ConceptDraft(
            path=f"services/{package}.md",
            concept_type="Service",
            title=package,
            body=text,
            citations=[],
            faithfulness_score=0.0,
            retry_count=0
        ))
    state["drafts"] = drafts
    state["status"] = "evaluating"
    return state

One instruction in that prompt does real work: do not invent dependencies you can't see in the code. Left alone, Haiku will confidently describe a dependency on a service that doesn't exist. That single sentence reportedly cut the hallucinated-dependency rate from roughly one file in six to nearly none in testing. Don't read that as a universal constant, because it depends on how much source goes into the prompt.

Step 4: the eval gate most people skip

A markdown file with a frontmatter type field can't know whether it's telling the truth. Nothing in the format prevents an LLM from writing a confident, well-formatted and wrong description of a service. Skip this stage and the system can silently corrupt an agent's understanding of its own codebase while looking exactly as trustworthy as a correct one.

Ragas does the scoring and DeepEval the pass/fail gate, because they solve different problems. Ragas produces a faithfulness score, meaning how well the draft is grounded in the real source, with no hand-written reference answer needed. DeepEval turns that number into something CI can act on.

from ragas.metrics import faithfulness
from ragas import evaluate
from datasets import Dataset
def score_faithfulness(draft: ConceptDraft, source_contents: str) -> float:
    eval_dataset = Dataset.from_dict({
        "question": ["Describe this service's responsibilities and dependencies."],
        "answer": [draft["body"]],
        "contexts": [[source_contents]],
    })
    result = evaluate(eval_dataset, metrics=[faithfulness])
    return result["faithfulness"][0]

from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric
def eval_gate(state: PipelineState) -> PipelineState:
    passed, failed = [], []
    for draft in state["drafts"]:
        source = read_package_source(state["repo_root"], draft["title"])
        score = score_faithfulness(draft, source)
        draft["faithfulness_score"] = score
        test_case = LLMTestCase(
            input="Describe this service.",
            actual_output=draft["body"],
            retrieval_context=[source]
        )
        metric = FaithfulnessMetric(threshold=0.75)
        try:
            assert_test(test_case, [metric])
            passed.append(draft)
        except AssertionError:
            draft["retry_count"] += 1
            failed.append(draft)
    state["drafts"] = passed
    state["failed_drafts"] = failed
    state["status"] = "publishing" if not failed else "needs_review"
    return state

The 0.75 threshold is a starting point and no law. It was raised from 0.6 after two drafts with subtly wrong dependency claims got through. A team adopting this should expect to spend a week watching its own false-positive and false-negative rates before it trusts the figure.

Step 5: wiring the graph with DynamoDB checkpoints

from langgraph_checkpoint_aws import DynamoDBSaver
checkpointer = DynamoDBSaver(
    table_name="codebase-memory-checkpoints",
    ttl_seconds=60 * 60 * 24 * 7  # expire checkpoints after a week
)
graph = StateGraph(PipelineState)
graph.add_node("scan_diff", scan_diff)
graph.add_node("draft_concept", draft_concept)
graph.add_node("eval_gate", eval_gate)
graph.add_node("relink", relink)
graph.add_node("publish", publish)
graph.set_entry_point("scan_diff")
graph.add_edge("scan_diff", "draft_concept")
graph.add_edge("draft_concept", "eval_gate")
graph.add_conditional_edges(
    "eval_gate",
    lambda s: "relink" if s["status"] == "publishing" else END,
    {"relink": "relink", END: END}
)
graph.add_edge("relink", "publish")
graph.add_edge("publish", END)
pipeline = graph.compile(checkpointer=checkpointer)

The choice of DynamoDB isn't arbitrary. The pipeline executes inside CI, where workers get killed mid-run more often than people admit. A checkpointed graph resumes at the last finished super-step rather than redrafting five files just because the sixth timed out.

Step 6: seeing what it costs

The pipeline wasn't trusted until each node's token use and duration were visible. Langfuse's @observe() decorator made that a five-minute addition and not a redesign.

from langfuse.decorators import observe
@observe()
def draft_concept(state: PipelineState) -> PipelineState:
    # same body as above, now traced automatically
    ...

Self-hosting needs only a docker-compose file, which counts if your code, even the diffs inside prompts, has to stay inside your network:

git clone https://github.com/langfuse/langfuse.git
cd langfuse
docker compose up -d

After ClickHouse acquired the project, self-hosted Langfuse keeps traces in both Postgres and ClickHouse, and self-hosting appears no less solid than before. Worth confirming before wiring observability into production, since whether a tool will be dropped or put behind a paywall is a reasonable thing to ask.

The dashboard shows three things: token cost per commit, the faithfulness distribution over time and the retry rate. A slow downward drift in faithfulness points to broken source-reading logic and not to the model getting worse. A retry rate above roughly 10% has traced back to the prompt every time, never the model.

A local version with no cloud

Some teams can't send even diffs to a hosted model. Swap Ollama in for the draft node and the same graph runs with no other change to the architecture.

import ollama
def draft_concept_local(state: PipelineState) -> PipelineState:
    drafts = []
    for package in state["changed_files"]:
        contents = read_package_source(state["repo_root"], package)
        response = ollama.chat(
            model="llama3.1:8b",
            messages=[{"role": "user", "content": DRAFT_PROMPT.format(file_contents=contents)}]
        )
        drafts.append(build_draft(package, response["message"]["content"]))
    state["drafts"] = drafts
    return state

For embedding-backed search over the published bundle, useful once there are more than a hundred concept files and index.md alone isn't enough, nomic-embed-text via Ollama is the model most local RAG setups already use, so no new dependency has to be justified.

+------------------+------------------------+---------------------------+
| Component        | Cloud version          | Local version             |
+------------------+------------------------+---------------------------+
| Draft model      | Bedrock Claude Haiku   | Ollama llama3.1:8b        |
| Embeddings       | Bedrock Titan Embed    | Ollama nomic-embed-text   |
| Checkpoint store | DynamoDB               | SQLite (LangGraph built-in)|
| Observability    | Langfuse (self-hosted) | Langfuse (self-hosted)    |
+------------------+------------------------+---------------------------+

In tests on the same source files, faithfulness from llama3.1:8b came out roughly 10 to 15 points under Haiku 4.5, mostly because it squeezes dependency descriptions harder. That gap is why the eval gate exists whichever model drafts. You don't want to learn about a quality difference from a confused agent three weeks later.

The git hook

#!/usr/bin/env bash
# .git/hooks/post-commit
set -euo pipefail

COMMIT_SHA=$(git rev-parse HEAD)
python -m codebase_memory.pipeline --commit "$COMMIT_SHA" --repo-root "$(git rev-parse --show-toplevel)"

Run it both ways: as a post-commit hook on a developer's machine and as a required CI job on the shared branch. Locally it flags drift before a PR exists. In CI it's the enforcement point, since some contributors never install the hook, and a hook that can't be enforced is merely advice.

A production reality check

Here's what is worth knowing before building this.

• Cost is real but modest. For a repo with about 40 commits a day touching two or three packages each, the Haiku draft pass costs somewhere in the low tens of dollars a month, and the eval pass roughly doubles that, since each draft gets scored against its source. That's a cheap price for not feeding wrong documentation to an agent.
• Latency is the trade-off nobody mentions. A full scan, draft, eval and publish cycle for a single-package commit takes 15 to 25 seconds in measurements on one setup. Fine as an async CI job, not fine as a synchronous block on committing, a mistake made in the first version before it moved to a background job.
• The gate will sometimes block a correct draft. Faithfulness scoring is imperfect, and an accurate but terse draft can fall below the threshold. Anything that fails twice goes to a human review queue and isn't silently discarded, which is the needs_review status in the state machine.
• It isn't worth building without several agents on your own codebase. All the value comes from agents consuming the bundle. An embedding-only version built first sat unused for two months because nothing read it.
• Package-to-file mapping won't transfer cleanly. Everything else is close to copy-paste. That one function is what you'll rewrite for your own layout, and it deserves real time and not a one-liner.

What to build next

At present the eval gate only scores faithfulness when a draft is written and never revisits a published file. The natural next step is a scheduled re-score of the entire bundle against today's source, to catch concept files that were accurate on the day and went wrong after three refactors with no redraft being triggered. That hasn't been built.

Evaluation thresholds are the number everyone tunes differently and almost nobody publishes, so if you build something similar, track where yours lands and whether it still holds after the first few weeks.