Why Coding Agents Still Struggle With Private Company Code

Public coding benchmarks test useful skills, but they say little about what happens inside a company's private production code. A benchmark called Real-SWE tries the second kind of test. Its builders, a small Y Combinator-backed startup called Specific Labs, licensed private code from real companies, extracted ten pieces of work that engineers there had already completed, and handed those jobs to eight coding assistants. The best one managed 38.8 percent.

One job opens with a billing emergency: invoices issued by the service are going out with no tax at all, and billing resumes on Monday. Across all assistants there were sixty-four attempts at it, and two worked. The task is called Tax jurisdiction, and it's one of the ten.

Why this code stays hidden

The founders published the benchmark recently, and it climbed the Hacker News front page with a few hundred points and well over a hundred comments. Real-SWE evaluates eight combinations of model and agent harness on ten tasks, with eight attempts per task. That makes 640 scored runs, and the headline score is the share of attempts that passed.

The underlying projects look like ordinary commercial software: a consumer events app with 200,000-plus users and a place in the App Store top 100, a fintech product that has ingested more than 100,000 bank statements, and a sales platform for enterprises. Agents get Docker and Kubernetes, three databases (Postgres, MySQL, MongoDB), Redis, GitHub and an emulated Amazon cloud, though each task only exposes the services its own workflow needs.

Some environments add Slack, Intercom, Google Drive, ClickUp and a Linear MCP server, which lets an agent read tickets straight out of Linear. When a detail never made it into the ticket, the agent has to dig it out of a thread or a document, much as that team's own engineer would.

The tax job alone runs against a NestJS service in TypeScript, two instances of the same external tax authority (a practice sandbox and the live system that files for real), and an InfluxDB ledger where completed sales are recorded.

Tasks use the Harbor format, and verification stays hidden until grading time. That's meant to lower the odds of an agent tuning for an exposed test and not for the actual requirement. The verifiers come from test suites those companies already run, in some cases verbatim.

The authors explain their motive with a striking claim: that 99 percent of tokens in real enterprises are hidden from frontier models. Because these tasks come from private software and not public GitHub projects, the models can't have seen the code, or its secrets, during training.

The benchmark's own numbers show a median instruction of 1,742 characters, which is shorter than FrontierCode (2,056) or DeepSWE (1,975). Yet the reference fix touches 11 files in the median case, where those two touch 6.

Results swing sharply from task to task

A single overall score masks two very different groups: tasks that agents solve again and again, and tasks they fail every time. Multi-region sweep resolved 67.2 percent of the time, API keys and environments 65.6, entitlement overage lines 50 and customer identity migration 40.6.

The tail of the list is grim. Tax jurisdiction sits at 3.1 percent and linearizable scan at 4.7, with S3 datastore measurement (10.9), API token metering (12.5) and billing schedule migration (14.1) not far above. On analytics stream reducer nobody got anywhere: zero passes in sixty-four rollouts.

On tax jurisdiction, the only two successes came from two different models, and each of those models failed the same job on its other seven tries. On multi-region sweep, the easiest on the board, three models went eight for eight while two others landed it twice. Averaging that to 67.2 percent hides one group that never missed and another that almost never got it.

One of the authors said on Hacker News that the low rates reflect tasks "that real engineers worked on for weeks".

The full ranking:

A score doesn't explain the failure

Every failed rollout carries one of five labels, borrowed from the taxonomy in DeepSWE:

• Unverified assumption. Instead of checking the workspace, the agent guesses at how the system behaves.
• Missed requirement. Something the instruction asked for never shows up in the change.
• Integration error. The approach is sound, but it's hooked into the surrounding system incorrectly.
• Regression. The change breaks something that previously worked.
• Wrong file. The edit goes into code that the running application never reaches, a throwaway script for example.

Each percentage uses the model's own failed runs as its denominator.

Grok 4.6 had 61 failures, two thirds of them (67.2 percent) missed requirements. Gemini 3.8 Flash had 55, and 49.1 percent of those were integration errors. GPT-5.6 Sol's biggest group is unverified assumptions, 43.3 percent of 67 failures, whereas Fable 5.1's 49 failures spread more evenly: about 37 percent are missed requirements and about 35 percent integration errors.

Two models, Grok 4.6 and Muse Spark 1.3, finished level at 23.8 percent yet fail differently. Grok's cluster around missed requirements, Muse Spark's around integration errors at 41 percent. Regressions never exceed 10.9 percent and are zero for three of the eight configurations. The wrong-file class appears just twice: GLM 5.3 (7 percent) and Kimi K3 (3.1).

A caution applies. The labels are published, but the process behind them isn't described well enough to treat the percentages as final evidence. The public write-up doesn't make clear whether human engineers, an automated model or a mix did the classification.

Longer runs weren't more reliable

The rollouts were also split by wall-clock time. Under ten minutes, 70 of 98 failed, or 71.4 percent. At ten minutes or more, 398 of 542 failed, or 73.4 percent.

Token use tells a related story. Grok 4.6 averages 3,000 to 20,000 output tokens on eight of the ten tasks, yet it produces 261,000 on linearizable scan and 315,000 on analytics stream reducer, the two lowest-scoring tasks. High token use can be a warning sign, especially when it sits far above a repository's normal baseline. When a run emits several times its expected output, inspect the trajectory or step in, instead of letting a stalled attempt burn budget.

The priciest agent wasn't always the best

The estimated cost of one rollout runs from $2.50 (Gemini 3.8 Flash) up to $6.96 (Fable 5.1), and the ordering hardly tracks the leaderboard. GPT-5.6 Sol costs $2.65 and ends up last, while the cheapest option, Gemini 3.8 Flash, takes third place. All of these are benchmark estimates and not a universal price ranking, since real spend depends on pricing, settings, task length, retries and tool use.

Fable 5.1 costs 2.78 times as much as Gemini 3.8 Flash and resolves 7.6 percentage points more. Whether that's worth it depends on what a failed task costs. On this set, a failure is a billing bug, a customer taxed when they were exempt, or a filing that doesn't reconcile at quarter end. For a UI tweak the sums look entirely different.

Much of the gap comes from token volume. GLM 5.3 averages 117,000 output tokens per rollout and GPT-6 Astra 24,000, nearly a five-fold difference on the same ten tasks. Two figures come with warnings from the benchmark's authors: Grok 4.6 at $3.44 and Kimi K3 at $3.90 are missing some usage data, so the true cost may be higher. Read them as floors.

Same harness, seventeen points apart

Every entry pairs a model with its own vendor's tooling, which is why the benchmark describes what it scores as "model-and-harness combinations rather than models in isolation". Two harnesses appear twice, though, giving a rough comparison. With Codex CLI, GPT-6 Astra resolves 33.8 percent against 16.2 for GPT-5.6 Sol, a 17.6-point gap. With Claude Code, Fable 5.1 reaches 38.8 and GLM 5.3 reaches 28.8, ten points apart.

Still, ten tasks and eight configurations make a small board, and every score carries a published confidence interval. Each model ran 80 rollouts, so Fable 5.1's 38.8 percent is 31 passes and GPT-5.6 Sol's 16.2 percent is 13. If a single task flips for one model, its score shifts by several points. Nobody should pick a vendor from a board this size, and the authors don't ask anyone to. What the exercise does show is that public scores may be an incomplete guide to performance in a private codebase, and it attaches five practical failure classes to that difference.

Run the five labels on your own code

The taxonomy is the part that travels. You can apply it to your own repository whenever you like.

Take the last twenty pull requests an agent opened, and give each failed or heavily edited one a single class from the five. Count them. The distribution is the finding, and it won't match whatever model you read about this week, since it depends on your codebase and ticket-writing habits as much as on the model. Whatever class turns out to be the biggest, put a gate against it.

If unverified assumptions dominate, require a reading log. Ask the agent to record which files it opened before changing anything, and reject the change if the list omits the file where the behaviour lives. Missed requirements usually trace back to the ticket. Turn the instruction into a list of acceptance criteria and tick them off individually, since an instruction with five requirements buried in a long paragraph is the shape behind Grok's 67.2 percent. Integration errors mean tests pass on the unit but fail on the call chain, so test along the route the live application follows, starting from its entry point. A wrong-file failure is rare enough that one occurrence deserves chasing: check that each modified file can be reached from code the application actually invokes.

From then on, follow one number: the share of agent pull requests that clear review first time, broken down by failure class. It shifts when the gate tightens and shifts again when the model underneath changes.