Context Engineering for AI Coding Agents: Why Bigger Windows Fall Short

Bigger context windows didn't fix coding agents

Intelligence isn't what holds AI coding assistants back anymore. The weak spot is how they store, fetch and connect information across a whole codebase.

When million-token windows arrived, the pitch was tidy: the model can finally take in your entire repository. Anyone who has pointed ChatGPT (Codex), Cursor, Claude Code or Gemini (Antigravity) at a real project knows how that plays out. Ask for a bug fix in AuthService.ts and the model forgets a helper that lives twenty files away. Add a new API route and it writes code against an outdated version of the interface. The window is enormous and the answer is still wrong.

The model isn't too small. The context is badly managed.

Token budget matters more than token limit

A context window is a ceiling, not memory you can fill freely. A mid-sized production repository holds a lot at once:

• source code spread across hundreds of files
• git diffs from changes in flight
• stack traces and logs
• API documentation
• configuration files
• READMEs and architecture notes

Cramming all of it into the prompt is costly and mostly pointless. Compare it with RAM in an operating system: a laptop with 32 GB doesn't load the whole disk into memory just because it could.

Coding agents behave the same way and manage a token budget. The question stops being what fits and becomes what deserves the tokens.

What gets priority

At the top of the list sits the file being edited, along with recent git changes, the dependencies it imports and the functions involved in the error. At the bottom you'll find unrelated documentation, old logs, unused modules and far-off corners of the repository.

Deciding what reaches the model, rather than just enlarging the window, is what people have started calling context engineering.

Why large projects produce more hallucinations

Coding hallucinations are rarely random. Usually the model got a fragment of the picture. Take this sequence. First, UserService.ts is loaded:

`UserService.ts` is loaded.

Second, PermissionManager.ts never makes it in:

`PermissionManager.ts` is not loaded.

Third, the model fills the hole with permission logic it made up:

The model invents permission logic.

From where the model sits, information that's absent looks identical to information that doesn't exist. Long-context research adds a second problem: models can have trouble using relevant details buried in the middle of a very long prompt, an effect called Lost in the Middle. So bigger windows take the edge off some failures but don't remove retrieval mistakes.

Think of a repository as a graph

This mental model changes how you approach the problem. A repository isn't a heap of files. It's a graph. Nodes are the meaningful things: files, classes, functions, APIs and database models. Edges are relationships such as imports, function calls, inheritance and data flow.

Now give the agent a task like "fix authentication timeout." It doesn't need to read 400 files. It walks the graph and picks up the neighborhood around authentication, which costs far less than loading everything.

Follow dependencies, not just keywords

Classic retrieval-augmented generation looks for chunks that are semantically close to the query. That helps, but code carries structure that prose doesn't. A better pipeline runs like this:

User query -> Semantic search finds relevant functions -> Graph traversal expands connected dependencies -> Context optimizer compresses everything into the token budget

So instead of isolated chunks, the agent gets connected context. This is the central idea in graph-based retrieval such as Microsoft's GraphRAG. Keep in mind that GraphRAG is a research architecture, and nobody has confirmed it runs inside commercial coding assistants.

A chat history is not memory

A common misconception treats conversation history as memory. Coding agents actually keep three layers:

1. Working context: the current prompt and the files that are active
2. Sliding window: recent conversation and edits
3. Retrieval memory: repository knowledge pulled in on demand

That's why Cursor or ChatGPT (Codex) can run a long coding session without carrying every earlier message in the prompt. They fetch what's relevant when it's needed instead of replaying the full conversation. Implementations differ between products, but the layered design is a common pattern across modern agents.

What a context pipeline looks like

The future isn't infinite windows. It's better pipelines. A context engine could do the following:

• Turn the repository into an AST.
• Build dependency graphs and call graphs from it.
• Create embeddings so semantic search works.
• Start at matching nodes and expand outward along dependencies.
• Compress what comes back before sending it to the LLM.

Breadth-First Search can collect nearby dependencies, and graph centrality or PageRank-style ranking can surface the files with the most influence. So building it takes ordinary software engineering at least as much as it takes model work.

Reading the right tokens

The race toward bigger windows matters, yet it won't be the whole answer. An assistant doesn't improve by reading more tokens. It improves by reading the right ones at the right moment. That's the move from prompt engineering to context engineering, and if agents are going to build production software, treating repositories as graphs instead of giant text files could turn out to be one of the more important architectural ideas for the next generation of assistants.

Where to read more

• Research paper on long-prompt weaknesses, Lost in the Middle (ACL 2024): https://aclanthology.org/2024.tacl-1.9/
• Microsoft GraphRAG documentation: https://microsoft.github.io/graphrag/
• GraphRAG GitHub repository: https://github.com/microsoft/graphrag
• The original RAG paper, NeurIPS 2020: https://arxiv.org/abs/2005.11401
• Building Effective AI Agents: https://www.anthropic.com/engineering/building-effective-agents