Scoring GraphQL Schema Design Debt with LLMs and a CI Ratchet

A GraphQL schema that dozens or hundreds of engineers change across many product domains drifts, no matter how good the style guide is. Linters catch the mechanical problems, but the expensive ones are judgment calls: a String that should have been an enum, a list that will grow without bound, a nullable field that never actually returns null. This article describes a two-part system for those cases: an LLM-assisted reviewer that stops new design debt at the pull request, and a scoring pass over the existing schema that turns old debt into a prioritized, trackable backlog enforced by a CI ratchet.

Why API quality becomes a systems problem

With a handful of engineers, a consistent API is mostly a matter of shared taste. People sit near each other, review each other's schema changes, and converge on the same patterns. As the organization grows, that stops working. New features ship constantly, older conventions coexist with newer ones, and decisions that felt obvious to the original team get applied differently by teams who never met them.

At that point, three questions need answers that do not depend on any one reviewer's attention:

• How do you keep API design consistent when many teams are editing the schema in parallel?
• How do you make sure new types and fields follow current best practices?
• How do you find the parts of the API that were designed before those practices existed?

The first two are about prevention. The third is about archaeology, and it is the one most governance efforts skip.

Where lint rules stop and judgment starts

A large share of API standards are mechanical, and static analysis handles them well. Naming conventions, use of deprecated fields, mandatory descriptions, and a consistent error shape are all yes-or-no properties of the schema: a field either follows the rule or it does not, and a linter can say which.

Other standards cannot be reduced to a clean rule. Typical examples:

• Should this String be an enum?
• Should this list be paginated?
• Should this Int be a custom scalar?
• Could this nullable field safely become non-null?
• Does this shape match how similar concepts are modeled elsewhere in the API?

None of these has a context-free answer. Returning a String, or even an untyped JSON blob, is sometimes correct. Deciding whether it is correct requires looking at three things together: the schema declaration, the resolver implementation behind it, and the intent behind exposing that data to clients. Only with all three can you judge what shape will serve clients best.

Organizations usually handle this with code review, office hours with the platform team, and written guidelines. That works, but it scales poorly. Delivery pressure shortens reviews, the platform team cannot look at every schema change in every repository, and best practices evolve faster than old APIs get revisited. The result is two related problems: preventing new design debt, and finding the debt that already exists.

Shifting left: an LLM reviewer for schema changes

The first half of the system encodes the API design guidelines into an automated code review agent. The aim is not to replace human reviewers but to give them a second pair of eyes on exactly the questions that slip through a normal pull request review. Having the platform team personally approve every GraphQL change across all repositories does not scale; pushing the preferred standards into an AI reviewer that runs everywhere does.

Because the agent sees more than the schema diff, it can reason about context rather than syntax. It reads the declaration, the implementation that backs the field, and the relevant policy text, and then raises specific questions. Two representative comments:

• A field named updatedAt is declared as String. If the resolver returns an ISO 8601 timestamp, it should probably use the dedicated ISO8601DateTime scalar instead.
• Company.employees returns a plain list. A company's workforce has no natural upper bound, so this field should return a paginated connection.

Neither of those is something a linter could flag reliably. A lint rule that says "fields ending in At must be date scalars" produces false positives and misses lastModified; a rule that says "all lists must be paginated" is wrong for a field returning the three supported currencies. The LLM can look at what the resolver actually does.

The payoff is timing. Catching these cases while the API is still being designed is cheap. Catching them after clients have adopted the shape means a deprecation cycle and a migration.

Looking backward: scoring the schema you already have

Prevention does nothing for the existing surface area, and in a mature API that surface is large. Some of it predates current standards. Parts of it encode compromises that made sense when they were made. And parts of it are just uneven, because separate teams modeled the same kind of concept in their own way. You need a way to look backward.

The second half of the system is a batch process that complements the static analysis tools already parsing the schema. Its pipeline:

1. Walk the schema domain by domain and select fields or types where subjective design judgment applies.
2. Gather the schema declaration together with the relevant implementation code.
3. Send that context to an LLM in a prompt that includes the written API design policies.
4. Ask the model whether the field appears to break one of those subjective practices.
5. Store the result as a score plus a written explanation.
6. Roll findings up by product domain or owning team.

Step 1 matters for cost and noise. There is no reason to ask a model about fields that a deterministic check has already classified; the LLM pass should only see the candidates where judgment is actually needed.

Why a 1-to-5 score beats pass/fail

Since these are judgment calls, forcing each finding into a binary verdict throws information away. Instead, each field gets a review score from 1 to 5:

• 1: the field looks appropriate as designed.
• 2: there is a weak signal, but it is probably fine.
• 3: a human should take a look.
• 4: the field likely violates the policy.
• 5: the field is a textbook case of the pattern to avoid.

To make that concrete: a String field that carries arbitrary user-written text should land near 1. A String field called errorCode whose resolver can only ever return one of three hardcoded values should land near 5, because it is an enum in disguise.

A graded score produces a much more useful signal than a flat violation list. Teams can start with the high-confidence 4s and 5s and still see the lower-confidence areas that may warrant a closer look. The middle of the scale has a second use: a cluster of 3s tells the platform team where the policy wording or the prompt is ambiguous, which is feedback for refining the judgment prompts until they produce more confident results.

If you build something similar, ask the model for structured output (a score and an explanation as separate fields) so results can be stored and aggregated without parsing prose, and keep the policy text and scoring rubric versioned alongside the prompt so score changes can be traced to rule changes.

Turning findings into action

Scores in a database change nothing on their own. Aggregating them by domain into a dashboard gives each owning team a concrete view of the API design debt in its area: not scattered anecdotes or one-off review comments, but a prioritized list of fields and types that may need migration.

The same data enables a ratchet in continuous integration. The point is not to fix everything at once, which is unrealistic for a large API with many production clients. The point is to make sure the situation never gets worse while the existing surface improves over time:

• New schema changes must meet the current standard.
• Existing issues are recorded as known debt rather than quietly ignored.
• As teams migrate or deprecate old patterns, the allowed threshold is tightened, so fixed debt cannot creep back.

Ratchets are a familiar pattern from lint migrations: record the current count of violations per area, fail the build if a change increases it, and lower the recorded baseline whenever someone fixes an instance.

This approach matters most for public or widely consumed APIs, where cleanup is gated by client migrations. The deliverable is not an instruction to delete every flawed field. It is a prioritized map of where the API no longer matches current standards, which teams can plan against.

Why an LLM is the right tool for this slice

LLMs are not flawless judges of API design, and the system does not treat them as such. Their strength here is narrower: reading code and schema together, comparing them against policy written in plain language, and producing a structured assessment for cases no static rule can express.

A static rule can tell you that a field returns a list. It cannot tell you whether that list grows with user input and therefore needs pagination. A model can read the resolver, compare it with the examples in the policy, and explain why the field does or does not fit the pattern.

That explanation is worth more than the number attached to it. When a field is flagged, the owning team needs to know why, so they can decide quickly whether the finding is real and, if it is, how to plan the migration. A score without a reason just creates another triage queue.

Limits you should plan for

LLM review does not replace API ownership or human design judgment, and it helps to be explicit about what remains:

• False positives still happen.
• Sometimes the implementation alone does not reveal the full picture, for example when a constraint lives in another service.
• Product constraints can make an imperfect shape the right trade-off anyway.
• The system does not migrate clients or make breaking changes safe. It finds debt; teams still have to plan and execute migrations carefully.

What it does provide is a scalable way to surface patterns that were previously limited by how much human review was available. Guidelines are encoded once, applied consistently across every repository, and the results give teams a factual starting point for design conversations.

Wrapping up

The system has two halves that share one idea. At pull request time, an LLM reviewer applies the design guidelines to new schema changes before clients depend on them. In batch, the same judgment scores the existing schema from 1 to 5, the scores roll up into per-team dashboards, and a CI ratchet keeps the total from growing while thresholds tighten over time. It is not fully automated governance, and it is not meant to be. It makes API quality visible enough that teams can act on it, and it gives the platform team a feedback loop for sharpening its own rules as the approach extends to more of the schema.