AI Engineering Measurements: Context Cost, Agent Evidence, and Interop
A map of executed AI measurements—MCP token overhead, prompt-cache breaks, agent audit gaps, model-switch friction—with methods and sample sizes attached.
This article is published in English.
Sources: Every AI Measurement We’ve Run: 43,600 Commits, 96 Releases, 9 MCP Servers
AI engineering commentary often states a belief and stops. A stricter habit is to take a popular claim, run the experiment it lacks, and ship the resulting figure with date, sample size, software versions, and archived raw data.
Figures on this map come only from runs that completed. Failed verification meant the result was dropped, with that failure noted.
Treat the sections below as an index that grows when new experiments finish.
What these measurements cover, and why three pillars
Pillar 1 — Context economics: what context actually costs
Teams purchase tokens they never authored; few publish counts of that overhead.
One study of MCP tool servers is summarized here: a measured MCP tool-server context study. Probing 116 tools over stdio burned 38,900 tokens before a human typed anything—19.4% of a 200k window. A single server drove 45% of that spend, and JSON Schema alone was 93.6% of that server’s share.
Separately, 96 published releases spanning nine servers were compared. Across 87 version transitions, 42 rewrote the tool block; 31 of those rewrites neither added nor removed a tool. Prompt caches broke on 48% of those releases. The tiniest textual break touched 34 tokens yet invalidated 36,320 cached ones.
Window capacity also depends on content type. A million-token window can hold about 4 MB of documentation prose but only about 1.8 MB of logs, because logs consume context roughly 2.2× faster—so cost is clearer in dollars per megabyte than in tokens alone.
Routing math shows a cheap model needs only about a 4% win rate to undercut a frontier-model bill. Related work on context rot explains why even a 2M-token window still loses the middle of long prompts.
Pillar 2 — Agent evidence: what agents record versus what you can prove
When an agent spends money overnight, reconstruction is often impossible.
Across six frameworks, audit logs carried a median of five fields though twelve were judged necessary. None stored a trust level or hash chain (so edits leave no scar) and none stored a stop reason.
Among CLAUDE.md files in the top 592 repositories, 57% contained no knowledge—only redirects. HEAD was probed for every repo; boundary cases were reviewed by hand both ways, and the estimate shifted twice before settling.
An AI-code survival census over 43,600 commits in eleven repos was repeated fifty times; twenty-one runs flipped the conclusion. Sampling blame on 200 of vllm’s 5,237 files instead of the full set turns a +12.2-point survival gap into zero. An earlier harness for the same question failed verification and was abandoned.
A line count of a large formal-proof artifact appears in a published line-count of Anthropic’s Fermat proof artifact: about 13 million lines total, 88.5% proof bodies, median step length 8 lines versus Mathlib’s 4.
Multi-agent “turf wars” on one repo were reproduced with no AI at all—classic file-locking dressed as agent conflict. Silent wrongness is another gap: one measured agent stayed incorrect on one ticket in fourteen for nine days with no dashboard alert.
Pillar 3 — Interop and switching cost: moving between models, protocols and tools
Marketing calls models and protocols drop-in. Diffs disagree.
Production model switches are not simple routers: only three parameters share identical spellings across the surfaces compared. MCP and A2A agent protocol specs share just two matching method names. Sandboxing alone also fails against the “lethal trifecta” breach pattern; isolation without complementary controls is not enough.
The most-read published piece
A local-AI playbook around Apple’s first 2nm chip asks whether a Mac can host a 70B model. With 512 GB of unified memory the answer is yes, and the write-up prices that setup against cloud alternatives.
How the measurement work is run, in four rules
- Executed work only. Posts state method, date, sample size, and versions; raw outputs are archived.
- Attack the number first. Bidirectional hand-checks plus a deterministic re-run have both destroyed attractive headlines.
- Name the blind spots. Nine servers are not an ecosystem; static analysis is not a runtime trace.
- No publish on failed measurement. Killed attempts are logged to avoid repeating the same mistake.
What’s next
Publishing cadence is about one measurement weekly. Current work asks how quickly an eval suite decays as the tested model improves: how many cases go quiet because capability truly rose, versus how many go quiet because the test itself rotted.