Home / Articles

Closed-Loop Evals for a Production Image Agent: What Uber Eats Presented

A case study of the evaluation design behind Uber Eats' photo enhancement agent: router recall, pass@K, drift diagnosis, autotuning and layered QA gates.

Publisher: REACTAPP.TOP7 min read1602 words

This article is published in English.

Sources: https://medium.com/@hello_62602/inside-uber-eats-closed-loop-eval-system-for-a-production-ai-agent-1bbe1206f5aa

Uber's ML team gave a conference talk at the AI Engineer World's Fair titled "Building Closed-Loop Evals for a Multimodal Agent at Scale." It's a rare, concrete look at how production evaluations work in a real system at real volume, and at the design choices that keep the system from degrading without anyone noticing. What follows is a case study of that system.

The problem

For many customers, a food photo is what tips the decision to order. For smaller, independent merchants on Uber Eats that photo is often weak: dim light, awkward framing, soft focus. The food isn't to blame. Professional photography takes time, skill and money that a small restaurant doesn't have.

An obvious remedy is to enhance the photo automatically with AI, but there's a snag. People distrust AI-generated food imagery. If the app shows something that looks unnaturally perfect, trust falls. The system therefore has to walk a narrow path. It must improve the picture without misrepresenting the dish, avoid flattening every merchant's photos into one generic look, and do all of this across a huge global marketplace where hand-editing is out of the question.

The stated goals were:

  • Stay faithful to the original dish
  • Improve quality only where it's genuinely needed
  • Keep the marketplace visually varied and not homogenized
  • Never publish anything that violates policy
  • Keep learning from new data
  • Do it all cost-efficiently at scale

Why an agent and not a fixed script

At one extreme sits a deterministic, rules-based photo editor. It's predictable and controllable, but brittle, because it can't cope with the range of dishes, cuisines and photo conditions in a worldwide marketplace. At the other extreme sits a fully unconstrained generative agent, which is flexible but offers no assurance that it won't invent ingredients or drift off brand.

Uber's design deliberately sits between the two. The agent gets real room to reason about each image, while every decision is wrapped in guardrails, QA gates and evaluation anchored to human judgment. Agency combined with constraint is the principle the whole pipeline is built on.

How the pipeline runs

  1. Understanding and routing. A multimodal model examines the photo along with text metadata such as dish name and description, and produces a structured description covering lighting, framing, whether the item fills the frame properly, and similar traits. A routing agent scores this against a rubric and makes a single choice: enhance, or skip and keep the original.
  2. The generation loop. A photo routed to enhancement enters a cycle of three steps. It produces an edit prompt specific to that image, performs the edit, and checks the result at a QA gate for faithfulness, unchanged regions, natural plating and preserved colors. When the check fails, the feedback is folded into the next prompt attempt. If failures continue past a capped number of rounds, the system doesn't push a poor edit through. It keeps the original photo.
  3. Final QA and publishing. A broader final pass runs before anything reaches a live menu.
  4. Logging. Each stage, for each image, adds to a single flat JSON record. This was built before any of the tuning systems described below, on the logic that a system can't be optimized unless it's fully observed, and a self-correcting loop needs a trail of data to learn from.

Evaluating the router as a classification task

The route-or-skip decision maps cleanly onto a confusion matrix:

  • True positive: a truly poor photo that is correctly sent for enhancement.
  • True negative: a truly good photo correctly left alone.
  • False positive: a fine photo that gets enhanced anyway. Compute is wasted, and a perfectly good picture risks being degraded.
  • False negative: a poor photo that slips by untouched. Uber's team treats this as the dangerous case, a silent regression nobody sees until a customer complains.

A missed bad photo costs more than an unnecessary enhancement, so routing is tuned with recall as the guardrail metric. Some wasted enhancement spend is accepted as the price of not letting bad photos through.

Two examples from the talk make the trade-off tangible. A nearly perfect cheeseburger photo was marked as technically below the bar and enhanced regardless, a precision miss that was pure wasted cost. In the opposite direction, a photo showing six chicken wings under a dish name reading "eight-piece wings" was passed over by the router. That was a recall miss, and a hazardous one, since sending it on to the enhancement agent risks the model hallucinating two extra wings to match the caption. Inventing detail to fit the text instead of what's visible is exactly what a downstream faithfulness check exists to catch, which shows why faithfulness checks matter in practice and not just on paper.

Evaluating generation as pairwise comparison

Once an image reaches enhancement, there is no fixed correct output. The question is whether the result is actually better than the input, judged on faithfulness, completeness and realism, with an answer of yes, no or unsure.

The number to watch is pass@K, the share of images that pass within K retries. A healthy curve climbs with K, since every failed attempt returns specific feedback to correct next time. One example from the talk involved sweet potato fries that failed the first QA for wrong portion size and unrealistic plating, then passed the second attempt once that feedback was applied.

The failure cases teach the most:

  • An edit that added shrimp which was never in the original dish fails on faithfulness outright.
  • An edit that removed a sauce cup at the edge of the plate fails on completeness.
  • A subtler case: the agent made a genuinely creative edit and the QA gate rejected it. Rather than applying a targeted fix, the model overcorrected and fell back to a bland, generic ceramic bowl. That passed QA but represented no real improvement.

That last case is reward hacking, meeting the letter of a metric while missing its intent. It's a reminder that a QA gate needs a target that captures "meaningfully better," and not merely "didn't trip an explicit check."

Two alignment loops: reaching v1, then staying correct

A first shippable version takes no special tricks. A dataset is assembled to span regions, dish types and photo quality levels, and human labelers grade it with an objective, bias-limiting rubric to form the golden set. The agent is run against that set. Clearing the guardrail number means release; missing it means more tuning.

Staying correct after launch is harder, because live traffic is messier than any fixed offline dataset can predict. The remedy is a steady sample of production items, graded by the same labelers on the same rubric, then set side by side with what the agent actually decided. Disagreements flow to a diagnoser agent, which locates the failing pipeline component and kicks off an automatic tuning pass for just that part, with no person in this step.

Two sub-agents make up the autotuner. Reflect studies the disagreements, discards the noise and names the systemic pattern behind them. Synthesize takes that finding, edits the affected agent's prompt or configuration, and re-tests the new version on the golden set before release. Only a passing version is saved as a new config version, and the following production run picks it up on its own. That is what stops a system that launched correctly from quietly decaying half a year on.

Three feedback loops running together

Drift correction is only one of three loops operating in parallel:

  1. The model loop. The automated drift and regression correction described above.
  2. The dogfooding loop. Internal teams and merchants give simple thumbs up or down along with free-text feedback before and during rollout, and that input also feeds the diagnoser.
  3. The marketplace loop. Genuine A/B tests against business metrics such as conversion and add-to-cart rate, broken down by geography, device and dish type, show where enhancement really moves results and where it doesn't.

The Swiss cheese idea

The closing concept ties it together. No single QA gate in this system is perfect: the routing rubric, the generation check and the final publish-ready check all have gaps. Stack several imperfect checks, though, and a bad output has to get through every one of them at once to reach a customer. The odds of that fall quickly with each added layer. It works like slices of Swiss cheese piled up, where each slice has holes but the chance of the holes lining up all the way through becomes small fast.

What there is to learn

This is what evaluation looks like once it outgrows a single offline test script:

  • Two metric families for two kinds of problem, classification for routing and pairwise comparison for generation.
  • One metric chosen deliberately as the guardrail, because false negatives cost more than false positives.
  • A pipeline that spots its own drift automatically, no human noticing first.
  • Layered QA built on the assumption that any individual check can and sometimes will fail.

None of it relies on exotic technology. It's disciplined system design around evaluation, applied at scale.