Making Two Local LLM Agents Debate: Grounding, Fake Consensus, Schema Bugs
Two local LLM agents on Ollama made to debate by evidence, not persona: NLI grounding, a fake consensus bug, attack vs assertion checks and required fields.
This article is published in English.
Sources: Two agents talking, and how to tell whether the conversation is real
Run on a single RTX 3060 Ti (8 GB) with qwen2.5:7b under Ollama, this experiment asks whether two local agents can hold a useful dialogue and how anyone could tell. It follows an earlier finding: two instances of one model aren't two opinions. A critic sharing weights, priors and failure modes with the researcher approved the researcher's hallucinations without friction.
Even so, building a two-agent dialogue is worth the attempt, since that conclusion left something out. Two copies of one model can't differ in judgment, but they can differ over facts if each holds different ones.
Difference by information, not personality
The usual way to make agents differ is to hand out personas: be optimistic, be skeptical, be the proposer, be the critic. The effect had already been measured. Telling a critic to be skeptical moved it from approving 13 of 13 findings to flagging 5 of 5, which is the same uniform behavior under a different label.
So the asymmetry has to be structural:
Agent A → read sources 1, 2. Cannot see sources 3, 4.
Agent B → read sources 3, 4. Cannot see sources 1, 2.
Now the first agent's question is a real one, since it doesn't know the answer, and the second agent's reply carries real information the first hasn't seen. Agreement between them also means something, because it spans two evidence sets. Prompting can't fake this. What goes into each context window enforces it.
Fluent messages that all failed
The first run produced six messages that were fluent, on topic and properly hedged. The grounding check rejected every one:
✗ A [answer] does not follow from its evidence (nli=0.03)
Our sources indicate that AI Agents are specialized, tool-enhanced
systems leveraging foundation models...
Yet the sentence appeared in that agent's evidence nearly verbatim. The entailment model still gave it 0.03. The trouble lay in what it was fed. NLI wants a clean proposition as the hypothesis, and the input was chatty phrasing along the lines of "Our sources indicate that..." or "Based on our sources, let's propose that...". That wrapper never appears in the quoted source, so the score collapses.
Earlier work on extracting findings hadn't run into this because a claim there is a bare sentence lifted from a paper. Dialogue is different. Its two readers want opposite things: the partner agent wants conversation and the checker wants clinical text. Splitting the message into two fields solved it. content goes to the other agent. claim is a single plain declarative sentence with no framing, and it's the only thing scored.
| Field | Text | Entailment |
|---|---|---|
content |
the hedged, conversational version of the point | 0.03 |
claim |
a bare statement that AI agents are specialized, tool-enhanced systems | 0.93 |
The model hadn't changed. The wrong string was being scored.
A consensus nobody reached
While those six messages were being rejected, the system printed this at the end of the conversation:
🤝 CONSENSUS (6 messages, 0 questions, 0 concessions)
AI memory architectures are evolving towards unified context layers...
No message had been grounded, since each contribution failed its evidence check, and still the system announced a confident consensus ready for pasting into a report.
The cause sat in the plumbing. The consensus step read the transcript, but the transcript() method filtered to grounded turns. It got an empty conversation, wrote a conclusion from the source quotes alone and labeled it as something two researchers agreed on.
This is the nastiest class of bug for a system like this. Nothing crashes and nothing refuses. The output reads well, looks tidy and carries a false history. Other failures in the project made noise, while this one passed for success. Three lines repair it: when no message survives grounding, converged is forced to false and the reason is printed. The deeper lesson outlasts the patch. Statements about process need the same verification as statements about facts, because "the agents agreed" describes the world and needs evidence.
Debating contradictions
A separate debate layer covers contradictions between two sources, with each agent defending one side from its own quote only. The first eight arguments all failed the grounding check.
The speaker's quote makes no mention of how AI memory systems are evolving, or of any potential for agents to collaborate.
Obviously that can't be entailed by the speaker's own evidence, because it describes the opponent's. A rebuttal was being scored against the wrong premise, which is a category error inside the checker. Rebuttals and assertions are different speech acts, so they get different tests:
- An assertion must be entailed by the speaker's own quote, measured with NLI.
- A rebuttal must address the opponent's specific claim, which is measured by word overlap and not by entailment.
Arguments with neither a grounded assertion nor a targeted rebuttal are still discarded. Saying "your evidence doesn't show X" is now a valid move and not an automatic failure.
Topics are picked without a model. Each pair of findings runs through NLI, and pairs with high contradiction probability become debates. At a threshold of 0.5, one pair got through at 0.517: a finding that AI agents are specialized, tool-enhanced systems, and another that memory systems are moving toward unified context layers. Those concern different subjects. Two agents still spent four rounds on a clash that wasn't there, which is the theater this layer was meant to prevent.
With the threshold raised to 0.7, the layer went quiet on that topic, finding zero conflicts among four papers on agent memory. That's correct, because the papers don't contradict each other. A debate system with nothing to debate is doing its job.
Optional fields don't exist
The new claim field started as optional with a default of "", and the model ignored it. All six messages fell back to conversational text and the run regressed. This was the fourth appearance of one mistake:
| Stage | Schema choice | Outcome |
|---|---|---|
| benchmark parsing | default_factory=list |
an empty {} still counted as a valid parse |
| critique output | field marked required yet an empty string was accepted |
about 30% of critiques came back blank |
| list with a default | the model emitted [] |
eight critiques disappeared |
dialogue claim |
default="" |
the model skipped it and all six messages failed |
The takeaway: an optional field is effectively absent, every time. To get a small model to produce something, the schema must reject output that lacks it.
Two schemas were needed, because a question can't carry a claim and a statement must. The next conversational need picks one: Question when it's time to ask and Statement otherwise, with claim and grounded_in both mandatory. A question looks like this:
❓ A: Could you provide more details on how multi-agent systems differ
from single-agent implementations, according to your sources?
And this is a round of answers:
✗ B: "Your sources do not directly compare multi-agent systems to
single-agent implementations." → rejected (nli=0.051)💡 A: Multi-agent systems often require more sophisticated memory
architectures and coordination protocols. (nli=0.985)💡 B: AI memory systems are evolving toward unified context layers for
efficient cross-agent collaboration. (nli=0.998)📋 A: [proposes joint conclusion] (nli=0.982)
📋 B: [proposes joint conclusion] (nli=0.540)🤝 CONSENSUS (6 messages, 1 question, 5 grounded)
Five of six messages were grounded. The one rejection is instructive. The second agent tried to characterize the first agent's sources, which it can't see. The grounding check caught that, not through any grasp of epistemics, but because a claim about one agent's papers can't be entailed by the other's.
One earlier run went better. Asked for specifics it lacked, the second agent admitted its sources didn't cover them, and the system reported no consensus along with the accurate reason: neither source supplied the examples that were requested. That's what the design was after from the start, an agent aware of what it doesn't know. A prompt asking for humility didn't produce it. Making unsupported statements fail did.
Cost
A turn takes 138 LLM calls at roughly two seconds each on the 3060 Ti. The dialogue accounts for 7, the debate for 5, synthesis for about 12, and the rest goes to extraction and criticism. It runs, but budget has become the constraint, not capability.
Lessons
- Differ by information. Different contexts yield different questions, while different instructions yield the same model in different costumes.
- Separate talk from measurable claims. One field is for the other agent and one for the checker, and a single string can't serve both.
- Verify process claims. "They agreed" needs evidence, and its absence made it the most dangerous output the system ever produced.
- Test attacks and assertions differently. Scoring "your evidence doesn't show X" against the speaker's own evidence is a category error.
- Accept an empty debate. Lowering the bar until it produces content produces theater.
- Make required fields required. Optional ones simply don't exist.
Whether any of this beats a single agent with a search tool remains unmeasured.