This article is published in English.
The GraphQL Query That Exhausted the Database Pool
One nested GraphQL document burned out a production database. Why rate limits, HTTP timeouts, and DataLoader failed — and the four bounds that finally contained it.
A single GraphQL POST knocked the API offline for almost an hour — and it was not malice
Pager noise started just after eight on a weekend night.
Postgres CPU pegged at full capacity. The pool had no free connections left. Clients saw gateway timeouts across the board. The public API was entirely unavailable during Sunday evening, which is nearly peak for that product’s usage pattern.
Traffic volume looked ordinary — even a little quiet for the day — so a sudden surge was unlikely.
Nothing had been released since midweek, so a bad deployment was also unlikely.
Roughly a quarter hour of digging surfaced the culprit, and it felt absurd at first glance.
Exactly one HTTP call. One POST /graphql had already spent about a minute and a half chewing database work and was still alive. That lone operation had already spent more DB time than the prior couple of hours of normal traffic.
What follows reconstructs the document shape, why existing controls missed it, and four bounds that landed a few days later. Near the end sits the worse case: how little effort a hostile actor would have needed against the same hole.
The query
Structurally accurate (though shortened), the document resembled:
query {
organizations {
members {
user {
organizations {
members {
user {
organizations {
members {
user { id, email }
}
}
}
}
}
}
}
}
}
Nesting walked seven levels around a cycle: orgs hold memberships, memberships point at people, people belong to orgs again.
Those edges are real and bidirectional, so modelling them that way is sound. Resolvers behaved. Each discrete SQL call was fine and quick on its own.
Damage came from combinatorial growth.
Typical accounts sit in roughly three orgs. Typical orgs list about fifty people. Expanding that walk yields on the order of:
- depth 1 → ~3 orgs
- depth 2 → ~150 memberships
- depth 3 → ~150 users
- depth 4 → ~450 orgs
- depth 5 → ~22.5k memberships
- depth 6 → ~22.5k users
- depth 7 → ~67.5k orgs
Nearly seventy thousand leaves at the bottom, each dragging more membership fetches. Expansion was still climbing when operators killed the work.
A short text document. No auth hole. No injectable string. Nothing a scanner marks. The schema simply obeyed the graph it published.
Accidental, not adversarial
Origin matters for the lesson.
The call rode an authenticated session of someone on staff. A mobile engineer had been poking the schema in Apollo Studio while sketching UI data needs, opening nested fields to see what existed.
They hit execute, watched the UI stall, blamed the network, and shut the browser tab.
Closing a tab does not abort server-side resolvers. The socket vanished; execution continued; the database kept walking tens of thousands of branches with nobody waiting on the response.
They only learned about the outage from Monday’s incident thread. Nothing hostile happened: they used the explorer the company handed them, against the schema the company shipped.
Why the existing shields missed
Controls were present. None matched this failure mode — and that mismatch is the point.
Per-IP request caps. Limits counted HTTP calls per minute. One call is one call. The limiter correctly waved it through.
Edge HTTP deadlines. A thirty-second balancer timeout fired; the caller saw 504. Backend SQL kept running because dropping the socket does not cancel work behind it. Clients were lied to; the server kept burning.
DataLoader. Teams often treat batching as the safety net.
Within a tick, DataLoader collapses duplicate entity loads and truly repairs classic N+1. At the deep membership layer, thousands of lookups folded into a few WHERE id IN (...) statements.
Folding twenty-odd thousand IDs into one statement does not make those rows free. Round trips shrink; cardinality does not; deeper levels still exist. Batching is an efficiency tweak, not a ceiling. Efficiency had been mistaken for a bound.
Login and identity. The caller was signed in. Identity answers who, never how expensive.
Per-field permissions. Every selected field was allowed for that user. Authorization succeeded. The bug was volume of legitimate graph walks, not forbidden data.
How ugly the same gap looks under attack
After recovery, an afternoon went into modelling hostile use of the identical hole. That modelling is why this piece exists.
Aliases let one document repeat a field with different arguments:
mutation {
a1: login(email: "target@company.com", password: "000001") { token }
a2: login(email: "target@company.com", password: "000002") { token }
a3: login(email: "target@company.com", password: "000003") { token }
# ... two thousand more
}
Still one HTTP request. Rate limits see one. A login-lockout counter that trips after five failures lived inside the resolver and correctly tallied thousands of tries.
So that password spray would have been stopped by accident — the counter happened to sit where work actually happens.
The broader pattern stayed open. Any costly resolver could be aliased hundreds of times inside a single request rate limits ignore: search, report jobs, third-party calls. Years of REST-shaped throttling faced an API that does not behave like REST.
Production also left introspection on. Anyone could pull the full type graph — every edge included — and craft maximal-cost documents without guessing.
Nobody had. Luck is not a control.
Four bounds that shipped afterward
A few days of engineering added the following, ranked by impact.
1. Depth limiting
First and simplest: refuse documents nested past a fixed ceiling.
import depthLimit from 'graphql-depth-limit';
const server = new ApolloServer({
schema,
validationRules: [depthLimit(7)]
});
Real client traffic was surveyed. Deepest honest operations stopped at five. The ceiling became seven — headroom for growth, rejection of the pathological case before resolvers start.
Validation rules inspect the parsed document pre-execution, so refusal is nearly free.
2. Query cost analysis
Depth is not enough. A shallow document that asks for ten thousand list items is still enormous.
Cost scoring weights fields, multiplies through list arguments, and rejects totals over a budget.
const server = new ApolloServer({
schema,
plugins: [
createComplexityPlugin({
maximumComplexity: 1000,
estimators: [
fieldExtensionsEstimator(),
simpleEstimator({ defaultComplexity: 1 })
]
})
]
});
Wiring is easy; choosing weights is the work. Scalars cost one. Lists cost first times child cost. Resolvers that hit third parties get manual weights such as fifty.
Two days of tuning against production logs got numbers roughly right. Roughly right was sufficient.
3. Alias and node count limits
Cap aliases per operation and total AST nodes.
Fifty aliases plus a node ceiling covered reality; no honest client approached them. Thousand-alias sprays become validation errors instead of thousand-fold resolver storms.
Packaged shields help. GraphQL Armor bundles depth, cost, alias, directive, and introspection controls. Greenfield teams should install and tune that pack before reinventing each piece.
4. Statement timeout at the database
The last line of defense — and the fastest win:
ALTER ROLE api_user SET statement_timeout = '10s';
Statements under the app role die after ten seconds. Not the HTTP socket — the SQL itself. That alone would have shrunk the outage from ~94s of DB work to ten, with zero GraphQL expertise.
Introspection in production was flipped off via config the same week — day-one hygiene that had been skipped.
Lessons for an earlier version of the same team
Three reminders.
Throttle cost, not merely request count. Per-minute HTTP counts are a REST habit. GraphQL can hide arbitrary work in one POST. If the meter only sees requests, there is no real meter. Meter cost.
Deadlines must kill the work. A thirty-second HTTP timeout looked protective and only hid damage from callers while backends kept spinning. Put timeouts where work runs — for Postgres, statement_timeout on the role.
DataLoader does not bound size. Batching erases N+1 and makes fat queries cheaper per round trip. It does not make them small. Efficiency and upper bounds are separate problems; ship both.
Checklist for production GraphQL
Verify these soon. Most take minutes.
- Introspection off in production? Otherwise the whole schema is public.
- Depth ceiling set? Measure deepest honest queries; sit slightly above them.
- Cost budget set? Depth alone misses wide lists.
- Alias ceiling set? Frequently missing; stops spray patterns.
- Database role
statement_timeoutconfigured? Minutes of work; contains this entire class. - Rate limits keyed on cost, not only request count? Count-only limits are theater.
That team started with one of six. They now run all six; four arrived in an afternoon.
The rule
Keep this distinction:
REST endpoints bound work by design. GraphQL lets clients bound work. Unless the server reinstates an explicit bound, the bound was not moved — it was deleted.
Prior defenses assumed servers decide how expensive a call can be. GraphQL hands that dial to whoever writes the document — powerful, and a common reason to adopt it. Responsibility must be taken back in code; the framework will not.
Nearly an hour of downtime began when a colleague pressed run in a studio. That is the friendly version. The hostile version needed only an account and a short thought; it never happened solely because nobody tried.
Confirm introspection settings first. It takes seconds, and many teams already know the answer.