Why Your Codebase Is a Graph: Tree-sitter Parsing for an AI Coding Assistant

Code is a graph, not a pile of text

Anyone who joins a new project knows the moment. You clone the repo and wonder where the payment retry logic lives. The reflex is one of these:

Ctrl + Shift + F

or

grep -r "retry" .

Sometimes that's enough, but often it isn't. The retry code might sit behind an interface, in a different module, inside a decorator, in a base class or behind a strategy pattern. Locating it takes more than string matching, because it depends on how pieces relate to each other.

Now ask Cursor "How does payment retry work?" and a couple of seconds later it highlights several files, explains the execution flow and even proposes improvements. It feels a bit like magic. The surprising part is that it isn't reading your whole repository, and it couldn't.

A common wrong mental model

Most developers first picture the tool like this:

Repository
      │
      ▼
Large Language Model
      │
      ▼
Answer

That sounds reasonable and can't actually work. Consider a mid-sized codebase:

12,000 Python files
≈ 2 million lines of cod

Even models with very large context windows can't process a whole repository efficiently on every prompt, and even if they could, the cost would be steep. Sending every source file, README, configuration, test and generated file would mean terrible latency and a huge bill. Worse, nearly all of those tokens would be irrelevant. If the question is about payment retries, there's no reason for the model to read image-processing code.

You don't read everything either

Watch how an experienced engineer finds things. Asked where authentication is implemented, nobody opens main.py and reads each file in order. A mental map forms right away, something like:

Request
   │
   ▼
API Router
   │
   ▼
Authentication Middleware
   │
   ▼
JWT Validator
   │
   ▼
User Service

You follow relationships, which means you traverse a graph. That distinction is a big one. Experienced engineers don't memorize millions of lines. They remember connections.

What a repository really is

A repository isn't a bag of files. It's a network of connected symbols. Take a small project:

src/
├── api.py
├── payment.py
├── retry.py
├── queue.py
└── logger.py

It looks like five unrelated files at first glance. Drawing the relationships tells a different story:

              api.py
                │
                ▼
         payment.py
           │     │
           ▼     ▼
      retry.py  logger.py
           │
           ▼
        queue.py

That picture explains more than the directory tree ever can. Every function call, import, inheritance link, interface implementation and dependency is an edge in the graph. Seen that way, AI coding assistants make more sense, since they navigate relationships and don't just search files.

Where text search fails

Look at a small example:

class PaymentProcessor:
def charge(self, order):
        return self.gateway.execute(order)

Where does the payment really happen? You can't tell. gateway might be Stripe, Razorpay, PayPal, a mock or a test double, and searching for "charge" won't resolve it. Add dependency injection and it gets worse:

processor = container.resolve(PaymentProcessor)

Now the implementation is chosen at runtime, and text search is nearly useless. Modern engineering leans heavily on abstraction, and ironically abstraction is what hurts naive search.

Code has structure

Programming languages aren't plain text. They're structured. Take this function:

def calculate_total(price, tax):
    return price + tax

You see characters, while a compiler sees structure, roughly like this:

FunctionDefinition
├── Name
│     └── calculate_total
├── Parameters
│     ├── price
│     └── tax
└── Return
      └── BinaryExpression(+)

That representation is an Abstract Syntax Tree, or AST. It keeps meaning and drops formatting. These two spellings:

return price + tax

and

return (
    price
    +
    tax
)

produce essentially the same AST. This is exactly why modern coding assistants start by parsing source code instead of treating it as raw text.

Tree-sitter

Many developer tools rely on Tree-sitter. Rather than scanning a file line by line, it builds a syntax tree that understands the language. Consider this code:

class PaymentService:
def retry_payment(self):
        pass

Tree-sitter doesn't see two lines of text. It sees something closer to:

Module
│
├── Class
│     └── PaymentService
│
└── Function
      └── retry_payment

That structure pays off later, because it lets you answer questions such as which class owns a method, which functions are public, which methods call a given function and which symbols are exported. Text search can never answer those reliably.

Parsing a repository

Time to build the first piece of the assistant. Install the dependencies:

pip install tree-sitter tree-sitter-python

Then parse a file:

from tree_sitter import Language, Parser
import tree_sitter_python as tspython

parser = Parser()
PY_LANGUAGE = Language(tspython.language())
parser.language = PY_LANGUAGE
code = """
class PaymentService:
    def retry_payment(self):
        print("Retrying...")
"""
tree = parser.parse(bytes(code, "utf8"))
print(tree.root_node)

The output looks like this:

(module
    (class_definition
        (identifier)
        (block
            (function_definition))))

It's not pretty, but it's useful. It tells you where every class starts, where each function ends, where imports sit and where decorators are defined, and nobody had to write a parser by hand.

Pulling out every function

Next, walk the syntax tree and collect all functions:

from tree_sitter import Parser, Language
import tree_sitter_python as tspython


parser = Parser()
parser.language = Language(tspython.language())

def extract_functions(node, source):
    if node.type == "function_definition":
        name = node.child_by_field_name("name")
        print(source[name.start_byte:name.end_byte].decode())
    for child in node.children:
        extract_functions(child, source)

source = open("payment.py", "rb").read()
tree = parser.parse(source)
extract_functions(tree.root_node, source)

Output:

retry_payment
refund
calculate_total

Note what happened here. The code didn't search for the keyword def. It asked the syntax tree for each function definition, and that's a meaningful difference.

Building an index

Instead of storing whole files, create a structured index with entries like this:

{
  "symbol": "retry_payment",
  "type": "function",
  "file": "payment.py",
  "class": "PaymentService",
  "start_line": 42,
  "end_line": 67
}

Do that for every function, class, interface, enum and constant, and an unstructured repository turns into searchable knowledge.

What exists so far

The pipeline built up to this point looks like this:

                Git Repository
                       │
                       ▼
              Repository Scanner
                       │
                       ▼
                Tree-sitter Parser
                       │
                       ▼
             Abstract Syntax Tree
                       │
          ┌────────────┴────────────┐
          ▼                         ▼
     Function Index           Class Index
          │                         │
          └────────────┬────────────┘
                       ▼
                Repository Graph

It doesn't resemble an AI system yet, and that's because it isn't one. It's a knowledge extraction pipeline, which is where every modern AI coding assistant begins.

Why start here

Many tutorials leap directly to embeddings and vector databases. That's like trying to build Google before any web pages exist. How well an assistant understands a repository before any machine learning happens largely determines how good it is. If the index is poor, no amount of prompt engineering rescues it. Bad input produces bad output.

What comes next

At this stage the assistant knows where each function and class is, but it doesn't understand meaning. The next problem is chunking. Most developers cut repositories into arbitrary blocks of 500 or 1,000 tokens, which sounds sensible and is one of the quickest ways to end up with a mediocre assistant. The better approach is semantic chunking: splitting along functions, classes, symbols and logical boundaries, so retrieval follows the way experienced engineers think about code. The result is a repository index made of meaningful units, ready for embeddings, vector search and smarter retrieval.