Home / Articles

When an AutoGen Agent Team Stalls: Measuring It and Escalating to a Supervisor

An AutoGen experiment with a deterministic shift-schedule referee shows why peer review fails, how to detect a stuck team, and why escalation needs a different model.

Publisher: REACTAPP.TOP9 min read2149 words

This article is published in English.

Sources: AutoGen ile Multi-Agent Eskalasyon: Ekip Takılınca Ne Olur?

A team of cheap agents can notice, by measurement, that it is stuck and hand the job to a stronger supervisor. Which supervisor counts as stronger turned out to be the surprise. What follows is a four-step AutoGen experiment: a single agent, a two-agent team, a deterministic validator and, finally, escalation.

Most multi-agent demos replay one scene. The agents fail to solve a problem, consult each other, and in the end a specialist agent steps in and finishes the work. It looks impressive until you read the code and find a line in the system message that says something like "if you can't solve it, ask the supervisor." The agent isn't asking for help because it failed. It is asking because it was told to. The whole scene is scripted.

Nothing is learned from that, because the hard question isn't how an agent asks for help. The hard question is how anybody finds out that a team is stuck. The experiment below was built so that getting stuck actually happens, and along the way two findings forced design changes. Both turned out to be the most valuable parts.

Taking the agents out of the judging seat

If you want to watch agents fail for real, they can't be the ones who decide what success means. So the task needs a result that a machine can check: a weekly shift schedule for a small store. There are seven days with two shifts each, and every shift needs exactly two people, which makes 28 slots to fill. Seven employees are available, with a combined capacity of 33 shifts. Six hard rules apply:

  1. Each shift has exactly two people.
  2. Nobody works two shifts on the same day.
  3. Whoever closes in the evening can't open the next morning, since 11 hours of rest are required.
  4. Weekly limits hold: five shifts for full-timers and three for part-timers.
  5. Nobody works on a day off.
  6. Every shift includes at least one cashier-certified person.

The difficulty sits where rules three and six overlap. One of the certified cashiers is a part-timer who can take only three shifts a week. Repair the rest rule and the cashier rule breaks, repair that and a limit gets exceeded.

The checker is a plain Python function with no LLM call anywhere inside it:

def dogrula(cizelge: dict) -> list[str]:
    """Cizelgedeki butun kural ihlallerini dondurur. Bos liste = gecerli plan."""

Every pass-or-fail verdict in the experiment comes from this function, and what the agents believe about their own work is irrelevant. The tests also prove that the problem is solvable, because a backtracking solver in the test suite finds at least one valid schedule. The agents never see that solver. Once the team gets stuck, nobody can claim the problem was impossible.

Step 1: a single agent and four violations

The simplest setup uses one AssistantAgent with no tools and no helpers:

planlayici = AssistantAgent("planlayici", model_client=istemci, system_message=...)
sonuc = await planlayici.run(task=brif())

With gpt-4o-mini as the model, the result was a well-formed, confident JSON schedule. The referee counted four violations. One employee was scheduled on a Monday off. Saturday evening had no certified cashier. A part-timer with a three-shift cap was given five shifts. The agent has no way to know any of this, since nothing in its setup lets it check its own answer.

Step 2: adding a teammate made it worse

The standard remedy is a reviewer next to the planner. In AutoGen that takes three lines:

ekip = RoundRobinGroupChat([planlayici, denetci], max_turns=6)
sonuc = await Console(ekip.run_stream(task=brif()))

The reviewer did its job, listing five violations, all of them under the heading of the 11-hour rest rule. The planner fixed them. When the referee then inspected the schedule, it found only one rest violation, so the reviewer had invented four. At the same time it had missed the day-off breaches, the missing cashier coverage and the weekly overruns entirely. The count rose to seven violations, worse than the lone agent.

None of this is surprising. The reviewer is also an LLM, so it carries the same weaknesses and burns twice the tokens. The team can also approve its own work: once the reviewer announces that the plan is approved, nobody pushes back. It is a student grading their own exam paper.

Step 3: correct feedback still isn't enough

The next move takes validation away from the LLM and hands it to the function. AutoGen accepts plain Python functions as tools, generating the schema from type hints and the description from the docstring:

denetci = AssistantAgent(
    "denetci",
    tools=[cizelgeyi_dogrula],
    max_tool_iterations=1,
    reflect_on_tool_use=False,   # <-- kilit satır
)

The setting reflect_on_tool_use=False is critical here. With True, the agent would summarize the tool output in its own words, the planner would read violations from that summary, and detail would be lost. With False, the validator's report enters the conversation word for word. The second key detail is the stopping rule:

bitis = TextMentionTermination("GECERLI:", sources=["denetci"]) | MaxMessageTermination(12)

Only the validator can produce the text GECERLI:, so the end of the conversation is decided by a deterministic signal instead of an LLM sentence. That fixes the "plan approved" problem from the previous step.

The outcome: the team reached its 12-message cap with nine violations. It does look worse. The feedback is accurate now, with no invented violations and no missed rules. But the small model fixes one rule while breaking another, and the process never converges. Correct feedback is necessary, yet insufficient. What's missing is a mechanism that measures progress and can declare that this team won't finish the job.

Step 4: measuring being stuck

The first question for escalation is whether the team is truly stuck or merely slow. Asking the agents is pointless, because expecting an honest self-report from a system that can't judge its own work repeats the mistake of step two. So the code measures, using two signals:

def takildi_mi(self, sabir: int = 2) -> tuple[bool, str]:
    son, onceki = self.denemeler[-1], self.denemeler[-2]

    # 1. Ayni ihlal seti tekrar ediyor -> dongude donuyorlar.
    if son.cizelge is not None and son.ihlaller == onceki.ihlaller:
        return True, "Ayni ihlaller iki tur ust uste tekrarlandi."

    # 2. Son turlarda ihlal sayisi hic azalmadi -> ilerleme yok.
    pencere = self.denemeler[-(sabir + 1):]
    if pencere[-1].skor >= min(d.skor for d in pencere[:-1]):
        return True, "Ilerleme yok."

    return False, ""

If neither signal fires, the team is improving, however slowly, and the loop continues. The check also lets the system escalate before the round budget runs out, which matters because every wasted round is wasted money. The orchestration is an ordinary Python loop. AutoGen keeps the team's chat history, so calling the same team object again means the team remembers earlier rounds:

for tur in range(1, MAX_TUR + 1):
    sonuc = await Console(ekip.run_stream(task=gorev))
    cizelge = son_cizelge(sonuc.messages)
    ihlaller = dogrula(cizelge)
    gunluk.ekle(Deneme(tur, cizelge, ihlaller))

    if not ihlaller:
        break
    takildi, sebep = gunluk.takildi_mi()
    if takildi:
        break
    gorev = sonraki_gorev(...)

What the escalation package carries

The team is stuck, so what goes to the supervisor? Asking the question again from scratch is easiest, but that is a retry and not an escalation. What a supervisor needs isn't the problem itself but what the team tried and why it failed. The package contains:

  • the problem brief,
  • the violation report for the schedule produced in each round,
  • the violation types that keep recurring, with counts (such as day-off: 3 times, limit: 3 times),
  • the team's final messages,
  • a clear task definition.

The recurring-violations line is a simple counter, but it works, because the supervisor sees at a glance which rule keeps locking the team up.

Finding 1: max_tool_iterations is a ceiling, not a guarantee

The supervisor received the validation tool with max_tool_iterations=8. The expectation was a self-contained loop of drafting, validating, fixing and validating again. What actually happened was this. The supervisor called the tool once, received a report of three violations, and then returned a completely different schedule as its final answer without validating it. That schedule was invalid.

The setting is an upper bound and never a lower one. The moment the model replies with text instead of a tool call, the loop ends. Writing "never answer without validating" into the system message didn't help either, since that line was already there. The fix was architectural: leave the last word to the referee and not to the model. The supervisor now runs inside the same outer validation loop as the team, and its answer isn't accepted until the referee passes it. Telling a model what to do doesn't make it happen. If you need a guarantee, you build the loop yourself.

Finding 2: a stronger model wasn't enough

The first supervisor was gpt-4o, over a worker team on gpt-4o-mini. It's the classic setup in which the cheap model works and the expensive one rescues. It didn't rescue anything. Over three rounds gpt-4o improved steadily from eight violations to five to four, never reaching zero, and produced 918 output tokens. After switching to the reasoning model o4-mini, the problem was solved in the very first round with zero violations, at the cost of 3,886 output tokens.

In hindsight this makes sense. The task is constraint satisfaction, which calls for systematic search. gpt-4o isn't meaningfully better than gpt-4o-mini at it, since both work the same way and one is just larger. The token gap is no accident either, because most of those 3,886 tokens are reasoning tokens. The price of thinking shows up in the cost table. The general lesson: escalation shouldn't mean a pricier model, it should mean a model that works differently. Measure which one helps on your own problem and don't assume.

The run finally succeeded, and the log shows how it unfolded:

--- Ekip - tur 1/3 (gpt-4o-mini) ----------------------------------------
[hakem] tur 1: GECERSIZ: 4 kural ihlali var.
--- Ekip - tur 2/3 (gpt-4o-mini) ----------------------------------------
[hakem] tur 2: GECERSIZ: 4 kural ihlali var.
--- Ekip - tur 3/3 (gpt-4o-mini) ----------------------------------------
[hakem] tur 3: GECERSIZ: 6 kural ihlali var.
[karar] Takilma tespit edildi: Son 2 turda ihlal sayisi 4 seviyesinden
        asagi inmedi; ilerleme yok.

--- Supervisor - tur 1/3 (o4-mini) --------------------------------------
[hakem] supervisor tur 1: GECERLI: Cizelge butun kurallara uyuyor.

There was no script behind it. The team really did get stuck, the stuck state was measured, and the supervisor solved the task.

The bill

At the end of step four, both models' spending is printed side by side:

Asama         Model        Girdi   Cikti  Maliyet $
------------  -----------  ------  -----  ---------
ekip (3 tur)  gpt-4o-mini  16.566  2.290  0,00386
supervisor    o4-mini       5.463  3.886  0,02311
------------  -----------  ------  -----  ---------
TOPLAM                     22.029  6.176  0,02697

The team's three rounds cost roughly $0.0039, while the supervisor cost about $0.0231, so escalation is a single line item worth about six times the team's whole effort. That table is the reason for the architecture. Hand everything to the supervisor and the bill is multiplied by six. Give everything to the cheap team and the work never gets done. The right answer lies between the two, and measurement is the only way to find it.

Takeaways

  • As long as you ask the agents whether they succeeded, the system is fake. A deterministic referee is the ground the whole architecture rests on. If your problem has no machine-checkable side, find one first.
  • Having one LLM audit another isn't enough. The reviewer invents its own hallucinations and doubles the cost.
  • Correct feedback is necessary and not sufficient. A small model can receive the right violation list and still fail to converge, so a progress-measuring mechanism is needed.
  • Escalation has four ingredients: a model that works differently, the failure history, the same deterministic validation loop and a system message that describes a systematic method.

How the code is organized

The experiment is split into four files meant to run in order, each opening with a note on what it teaches:

01_tek_agent.py      AssistantAgent, run(), TaskResult
02_ekip.py           RoundRobinGroupChat, max_turns, Console
03_dogrulayici.py    tools=[...], reflect_on_tool_use, deterministik termination
04_eskalasyon.py     dış döngü, takılma tespiti, eskalasyon paketi, maliyet defteri

The tests need no API key. AutoGen's ReplayChatCompletionClient returns model replies in sequence, so the two riskiest assumptions of the project can be verified for free. The versions used were autogen-agentchat 0.7.5 and Python 3.11. Small-model behavior varies from run to run, and gpt-4o-mini sits right at this problem's edge, sometimes finishing in four rounds.