阻止失控的AI智能体循环：面向生产环境的四项防护机制
控制流程转向、令牌与时间管理，通过实际测试验证目标，利用哈希工具检测乒乓循环现象，当自主智能体陷入僵局时安全回滚。
来源: Don’t Let Your AI Agents Loop Forever: An Engineering Guide to Termination Criteria
有一个自动化代理在语法修复循环中卡了12小时，最终产生了3,800美元的API费用，这是没人愿意经历的“成长仪式”。以下是为何要在代理系统中设置生产级停止规则的方法。
周五下午的场景
现在是周五下午5点，一个新的编程代理被放入测试环境。它的任务是检查拉取请求、修复出错的TypeScript类型并重新运行测试。提示语看起来没问题，笔记本电脑关闭，周末就此开始。
到了周六早上，PagerDuty和账单警报不断涌入电话。该代理修复了user.service.ts中的类型错误，却因此破坏了auth.service.ts。修复那个文件又导致了第一个文件的故障，这两个文件不断互相破坏。经过1,400次工具调用后，Token费用高达3,800美元，而且测试套件也遭到了损坏。
大多数工程关注点都集中在提示词、工具选择以及调度图上。在实际应用中，影响最大的决策是智能体如何停止运行，而非它如何启动。
什么是智能体循环
智能体循环是一种控制循环，其中大语言模型会感知当前状态、选择动作、调用工具、读取输出结果，然后决定是否继续执行。
┌──────────────────────────────────────┐
│ Agent Perception Loop │
│ (Perceive → Plan → Act) │
└──────────────────┬───────────────────┘
│
┌──────────────────────────┼──────────────────────────┐
▼ ▼ ▼
┌────────────────────┐ ┌────────────────────┐ ┌────────────────────┐
│ 1. Success Guard │ │ 2. Resource Caps │ │ 3. Progress Guard │
│ (Programmatic Test)│ │ (Tokens/Turns/Time)│ │ (Loop/Hash Detect) │
└────────────────────┘ └────────────────────┘ └────────────────────┘
│
▼
┌──────────────────────────────────────┐
│ 4. Human Handoff / Safe Rollback │
└──────────────────────────────────────┘
如果其周围的框架缺乏分层且明确的终止规则，该智能体最终会陷入三种故障模式之一：
- 无限执行，在虚假或重复的工具调用中不断循环。
- 提前退出，在任务尚未完成或已悄然失败时便宣布胜利并停止运行。
- 预算耗尽，在多步骤任务尚未完成之前就耗尽了所有令牌额度。
有四种停止标准可用于处理这些情况，每个智能体在上线前都需要具备这四种标准。
1. 成功标准：确定性验证目标达成
隐患所在。绝不能仅通过询问大语言模型是否已完成任务就接受其自身的判断。模型往往存在完成偏误和讨好倾向，即便构建过程出错或输出文件为空，它们也常常会报告任务已成功完成。
解决方案。将模型输出视为未经验证的假设，让循环通过某种确定性方式来检查实际状态：
- 在子进程中运行
npm test或pytest，并读取退出码 - 使用具有严格退出码的类型检查与代码规范检查工具，例如
tsc --noEmit - 通过 Zod 或 Pydantic 模式解析输出
import { execSync } from 'node:child_process';
export interface VerificationResult {
success: boolean;
message: string;
stepFailed?: string;
}
/** execSync throws on non-zero exit; diagnostics may land on either stream. */
function runOrCapture(command: string, cwd: string): string | null {
try {
execSync(command, { cwd, stdio: 'pipe' });
return null;
} catch (err: unknown) {
const e = err as { stdout?: Buffer; stderr?: Buffer };
const out = e.stdout?.toString() ?? '';
const errOut = e.stderr?.toString() ?? '';
return [out, errOut].filter(Boolean).join('\n') || String(err);
}
}
export class GoalVerifier {
public static verify(workspacePath: string): VerificationResult {
const steps: Array<[string, string]> = [
['tsc', 'npx tsc --noEmit'],
['npm_test', 'npm test'],
];
for (const [stepId, command] of steps) {
const failure = runOrCapture(command, workspacePath);
if (failure !== null) {
return {
success: false,
message: `\`${command}\` failed:\n${failure}`,
stepFailed: stepId,
};
}
}
return { success: true, message: 'All typechecks and tests passed cleanly.' };
}
}
原则很简单：如果人工审核员会使用某个命令来检查拉取请求，那么这个命令就应该作为循环的退出条件。
2. 资源与预算上限：刚性限制
隐患。当智能体遇到边缘情况或意外的工具错误时，它的默认做法是通过微调提示词来重试。由于在引擎层面没有上限限制，一个陷入死循环的智能体可能在几分钟内就消耗数百万个标记。
解决方案。在三个独立维度上设置刚性上限：轮次、标记数以及实际耗时。需分别统计轮次和标记数，因为一个轮次可能涉及多次模型调用，若将两者混为一谈，就会悄悄增加轮次计数。
export interface ResourceLimits {
maxTurns: number; // e.g. 10 iterations
maxTotalTokens: number; // e.g. 100_000 input + output
timeoutMs: number; // e.g. 120_000 (2 minutes)
}
export class ResourceGuard {
private readonly startTime = Date.now();
private totalTokensUsed = 0;
private currentTurn = 0;
constructor(private readonly limits: ResourceLimits) {}
/** Call once per loop iteration, before the model call. */
public beginTurn(): void {
this.currentTurn += 1;
}
/** Call for every model call, including retries inside a turn. */
public recordUsage(tokens: number): void {
this.totalTokensUsed += tokens;
}
public getTurnCount(): number {
return this.currentTurn;
}
public checkShouldTerminate(): { terminate: boolean; reason?: string } {
if (this.currentTurn >= this.limits.maxTurns) {
return {
terminate: true,
reason: `Exceeded turn cap (${this.limits.maxTurns})`,
};
}
if (this.totalTokensUsed >= this.limits.maxTotalTokens) {
return {
terminate: true,
reason: `Exceeded token budget (${this.totalTokensUsed}/${this.limits.maxTotalTokens})`,
};
}
const elapsed = Date.now() - this.startTime;
if (elapsed >= this.limits.timeoutMs) {
return {
terminate: true,
reason: `Wall-clock timeout reached (${elapsed}ms/${this.limits.timeoutMs}ms)`,
};
}
return { terminate: false };
}
}
上限仅是保障措施，而非设计考量。如果智能体因达到maxTurns限制而频繁终止运行，问题并不出在上限上，而在于任务拆分方式。
3. 进度监控：识别陷入僵局或偏离目标的智能体
有三种模式反复出现：
- 乒乓式循环。智能体先使用工具A，遇到错误X，再使用工具B尝试修复，结果又出现错误Y，最后仍用相同的输入重新运行工具A。
- 混乱式操作。修改文件A会破坏文件B，修改文件B又会破坏文件A。工作区不断变化，但整体进度始终为零。
- 语义偏离。智能体被意外的代码检查警告干扰，花费五轮时间处理格式问题，却忘记了最初的目标。
为在 maxTurns 用尽之前发现死锁，需维护一个工具调用哈希值与工作区差异哈希值的滑动窗口。若出现重复的相同哈希值，说明智能体正在陷入循环。
import { createHash } from 'node:crypto';
export interface ToolCall {
name: string;
args: Record<string, unknown>;
}
export class ProgressGuard {
private readonly recentActionHashes: string[] = [];
constructor(
private readonly windowSize = 5,
private readonly repeatThreshold = 3,
) {}
/** Stable stringify: key order must not change the hash. */
private hashToolCall(call: ToolCall): string {
const args = JSON.stringify(call.args, Object.keys(call.args).sort());
return createHash('sha256').update(`${call.name}:${args}`).digest('hex');
}
/** Returns true when the same call has appeared `repeatThreshold` times in the window. */
public trackAndCheckStuck(call: ToolCall): boolean {
const actionHash = this.hashToolCall(call);
const priorOccurrences = this.recentActionHashes.filter((h) => h === actionHash).length;
this.recentActionHashes.push(actionHash);
if (this.recentActionHashes.length > this.windowSize) {
this.recentActionHashes.shift();
}
return priorOccurrences + 1 >= this.repeatThreshold;
}
}
并非所有的重复操作都应受到同等怀疑。三次只读调用通常无害，而三次写入操作则几乎从来都不是无害的。当有工具元数据可用时，应调整阈值以匹配实际情况，而非一视同仁地对待所有调用。
4. 人工介入与安全回滚
当循环因预算上限或死锁而终止时，最糟糕的结果就是系统出现部分故障。断路器应按顺序执行三件事：收集诊断信息（哪些工具失败以及最后出现的错误）、将工作空间恢复到已知状态，然后向相关人员提交清晰的总结报告以便进一步处理。
第二步需格外小心。执行 git reset --hard 后再使用 git clean -fd 的操作是不可撤销的。git stash --include-untracked 更为安全，因为它能保留暂存区中的内容以便日后查看，同时也仍可随时丢弃。建议将彻底重置的操作留到临时测试环境中使用。
import { execSync } from 'node:child_process';
import { writeFileSync } from 'node:fs';
import { join } from 'node:path';
export class AgentEscalationRequiredError extends Error {
constructor(message: string, public readonly reportPath?: string) {
super(message);
this.name = 'AgentEscalationRequiredError';
}
}
export class AgentCircuitBreaker {
constructor(
private readonly workspaceDir: string,
private readonly reportDir: string, // keep reports OUTSIDE the workspace
) {}
public handleAbort(reason: string, history: unknown[] = []): never {
console.error(`[CIRCUIT BREAKER] Terminating agent loop: ${reason}`);
// 1. Park workspace changes recoverably.
try {
execSync('git stash push --include-untracked -m "agent-abort"', {
cwd: this.workspaceDir,
stdio: 'pipe',
});
} catch (err) {
console.error('git stash failed during abort; workspace left as-is:', err);
}
// 2. Write a diagnostic trace for human review.
const reportPath = this.writeFailureReport(reason, history);
// 3. Signal the orchestrator.
throw new AgentEscalationRequiredError(`Agent failed safely. Reason: ${reason}`, reportPath);
}
private writeFailureReport(reason: string, history: unknown[]): string {
const reportPath = join(this.reportDir, `agent_failure_${Date.now()}.json`);
writeFileSync(
reportPath,
JSON.stringify(
{
timestamp: new Date().toISOString(),
workspace: this.workspaceDir,
reason,
historyLength: history.length,
history: history.slice(-10),
},
null,
2,
),
'utf-8',
);
return reportPath;
}
}
案例研究：开放式应用生成器
修复漏洞时 có 一套固定的测试用例可供验证。而更棘手的情况则是那些开放式、主观性的任务描述，例如：在本地构建一个能够管理任务的现代网页应用，让它自动运行直至任务完成，且界面要美观、功能完备，同时由开发者自行决定何时停止运行。
“美观”和“完成”这类表述其实非常模糊。如果不加约束，智能体要么只生成一个简单的 index.html 文件就停止，要么就会一直调整 CSS 边距却永无止境。
┌──────────────────────────────────────────────────────────┐
│ Vague prompt ("build a modern web app locally") │
└────────────────────────────┬─────────────────────────────┘
▼
┌──────────────────────────────────────────────────────────┐
│ Phase 1: Dynamic spec synthesis (`ac-matrix.json`) │
└────────────────────────────┬─────────────────────────────┘
▼
┌──────────────────────────────────────────────────────────┐
│ Phase 2: Multi-agent execution loop │
│ (Coder agent + design critic + headless E2E verifier) │
└────────────────────────────┬─────────────────────────────┘
│
┌─────────────────────────┼─────────────────────────┐
▼ ▼ ▼
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ Gate 1: Goal │ │ Gate 2: Resource │ │ Gate 3: Progress │
│ verification │ │ caps (turns/ │ │ guard (deadlock/ │
│ (build/E2E/ACs) │ │ token budget) │ │ repetition) │
└─────────┬────────┘ └─────────┬────────┘ └─────────┬────────┘
└───────────────────────┼───────────────────────┘
▼
┌──────────────────────────────────────────────────────────┐
│ Gate 4: Safe exit OR circuit-breaker rollback │
└──────────────────────────────────────────────────────────┘
首先制定规范
在第一步中，即在编写任何代码之前，智能体会生成一个明确的验收标准矩阵，并将其保存为ac-matrix.json。该矩阵将标准分为三类：
- 功能标准，例如本地SQLite或JSON持久化存储、CRUD操作以及深色模式切换功能。
- 工程标准，例如没有TypeScript相关警告和lint错误，以及启动时没有控制台错误的开发服务器。
- 设计标准，例如响应式布局、统一的设计规范、空状态处理以及简洁的交互设计。
用于生成该矩阵的同一模型日后会依据它来评估自身表现，因此应将其视为范围契约而非客观标准。它的真正价值在于能够在智能体还有机会改变目标之前就提前锁定这些标准。
职责分工
为避免某个智能体将自己完成的工作评为完美，需将任务分开处理：
- 编码子智能体负责构建功能、路由及样式处理。
- 无头浏览器验证器启动开发服务器，使用Playwright截取屏幕截图并检查控制台错误。
- 设计评审子智能体依据设计系统标准，从层级结构、间距和对比度等方面审查用户界面。
框架结构
以下代码依赖于注入的组件（llmAgent、executeTool、runHeadlessBrowserCheck和runDesignCriticAgent），您可将其替换为自己的调度层。
import { execSync } from 'node:child_process';
import { readFileSync } from 'node:fs';
import { join } from 'node:path';
import { ResourceGuard, ProgressGuard, AgentCircuitBreaker } from './guards';
interface AcceptanceCriterion {
id: string;
description: string;
status: 'PENDING' | 'IN_PROGRESS' | 'DONE';
}
interface AcMatrix {
items: AcceptanceCriterion[];
}
export interface RunResult {
success: true;
turns: number;
summary: string;
}
export async function runOpenEndedWebAppGenerator(
userPrompt: string,
workspacePath: string,
reportDir: string,
devServerUrl = 'http://localhost:5173',
options = { maxTurns: 15, maxTotalTokens: 200_000, timeoutMs: 300_000 },
): Promise<RunResult> {
const resources = new ResourceGuard(options);
const progress = new ProgressGuard();
const circuitBreaker = new AgentCircuitBreaker(workspacePath, reportDir);
const acMatrixPath = join(workspacePath, 'ac-matrix.json');
let currentPrompt = userPrompt;
let designRetries = 0;
const maxDesignRetries = 3;
while (true) {
// GUARD 1: resource ceilings
const resourceCheck = resources.checkShouldTerminate();
if (resourceCheck.terminate) {
circuitBreaker.handleAbort(resourceCheck.reason!);
}
resources.beginTurn();
const turnResult = await llmAgent.step(currentPrompt);
resources.recordUsage(turnResult.tokensUsed);
// GUARD 2: deadlock detection (only meaningful when a tool was called)
let toolOutput = '(no tool call this turn)';
if (turnResult.toolCall) {
if (progress.trackAndCheckStuck(turnResult.toolCall)) {
circuitBreaker.handleAbort('Repeating tool call detected (stuck agent)');
}
toolOutput = await executeTool(turnResult.toolCall);
}
// GUARD 3: deterministic convergence check
const verification = await evaluateConvergence(workspacePath, acMatrixPath, devServerUrl);
if (verification.gateFailed === 'design') {
designRetries += 1;
if (designRetries > maxDesignRetries) {
circuitBreaker.handleAbort(
`Design gate never converged after ${maxDesignRetries} refinement passes`,
);
}
}
if (verification.isConverged) {
return {
success: true,
turns: resources.getTurnCount(),
summary: 'Web app built, tested, and design-reviewed cleanly.',
};
}
currentPrompt = `Tool output:\n${toolOutput}\n\nConvergence status:\n${verification.statusMessage}`;
}
}
interface ConvergenceResult {
isConverged: boolean;
statusMessage: string;
gateFailed?: 'build' | 'runtime' | 'acs' | 'design';
}
async function evaluateConvergence(
workspacePath: string,
acMatrixPath: string,
devServerUrl: string,
): Promise<ConvergenceResult> {
// Gate 1: build and typecheck
try {
execSync('npx tsc --noEmit && npm run build', { cwd: workspacePath, stdio: 'pipe' });
} catch (err: unknown) {
const e = err as { stdout?: Buffer; stderr?: Buffer };
const output = [e.stdout?.toString(), e.stderr?.toString()].filter(Boolean).join('\n');
return {
isConverged: false,
gateFailed: 'build',
statusMessage: `Gate 1 failed (build/typecheck):\n${output || String(err)}`,
};
}
// Gate 2: dev server and runtime health
const e2eResult = await runHeadlessBrowserCheck(devServerUrl);
if (!e2eResult.noConsoleErrors) {
return {
isConverged: false,
gateFailed: 'runtime',
statusMessage: `Gate 2 failed (console errors): ${e2eResult.errors.join(', ')}`,
};
}
// Gate 3: acceptance criteria fully complete
let acMatrix: AcMatrix;
try {
acMatrix = JSON.parse(readFileSync(acMatrixPath, 'utf-8')) as AcMatrix;
} catch {
return {
isConverged: false,
gateFailed: 'acs',
statusMessage: 'Gate 3 incomplete: `ac-matrix.json` missing or unparseable.',
};
}
const pending = acMatrix.items.filter((ac) => ac.status !== 'DONE');
if (pending.length > 0) {
return {
isConverged: false,
gateFailed: 'acs',
statusMessage: `Gate 3 incomplete: ${pending.length} ACs remaining (${pending
.map((a) => a.id)
.join(', ')})`,
};
}
// Gate 4: design audit (soft gate — see retry cap in the caller)
const criticVerdict = await runDesignCriticAgent(e2eResult.screenshots);
if (criticVerdict.score < 8.5) {
return {
isConverged: false,
gateFailed: 'design',
statusMessage: `Gate 4 incomplete (design ${criticVerdict.score}/10): ${criticVerdict.feedback}`,
};
}
return { isConverged: true, statusMessage: 'All four convergence gates passed.' };
}
请注意，这些评估标准并不相同。前两个，即编译器和浏览器，属于确定性系统，因为它们没有主观判断。而后两个则由模型来评估，这意味着它们的结果可能会一直来回波动。正因如此，设计评估标准需要设置自身的重试上限。没有尝试次数限制的主观阈值只不过是速度更慢的无限循环罢了。
可重复使用的主提示词
若要在 Claude Code、Copilot CLI、Cursor 或 Aider 中运行此流程，下面的模板会指导智能体自行编写验收矩阵，在开发周期中不断迭代，本地检查结果，评估设计质量，只有当应用开发完成时才会停止。
You are an autonomous lead software engineer, UX designer, and QA verifier. Your goal
is to build a production-quality web application locally, from scratch.
You must operate in a self-terminating agentic loop, running iteratively until the
application is complete, polished, functional, and verified.
================================================================================
1. TARGET APPLICATION SPECIFICATION
================================================================================
[DESCRIBE YOUR APP IDEA HERE — e.g. "A task management web app with local SQLite
persistence, a kanban board with drag-and-drop, priority tags, search/filter
controls, and a dark mode theme."]
================================================================================
2. EXECUTION PROTOCOL
================================================================================
PHASE 1 — DYNAMIC SPEC SYNTHESIS (TURN 1)
Before writing application code or installing dependencies:
1. Initialize the local project structure (e.g. Vite + React, Next.js, or Node).
2. Create `ac-matrix.json` in the workspace root defining explicit acceptance
criteria:
- Feature ACs: persistence, full CRUD, interactive components, error
handling, edge cases.
- Engineering ACs: strict TypeScript (`npx tsc --noEmit`), zero build
errors, zero linter warnings, dev server boots cleanly.
- Design ACs: visual hierarchy, responsive layout, dark/light toggle,
empty states, micro-interactions.
Format:
{
"project": "<app-name>",
"items": [
{ "id": "FEAT-1", "category": "feature", "description": "Local database persistence for tasks", "status": "PENDING" },
{ "id": "FEAT-2", "category": "feature", "description": "Drag-and-drop kanban re-ordering", "status": "PENDING" },
{ "id": "ENG-1", "category": "engineering", "description": "Clean TypeScript build, zero errors", "status": "PENDING" },
{ "id": "ENG-2", "category": "engineering", "description": "Dev server starts with 0 console errors","status": "PENDING" },
{ "id": "DSGN-1", "category": "design", "description": "Responsive UI with dark/light mode", "status": "PENDING" }
]
}
Once written, treat `ac-matrix.json` as frozen scope. Do not delete or weaken an
AC to make a gate pass. If an AC turns out to be genuinely infeasible, mark it
BLOCKED with a reason and surface it in the final summary.
PHASE 2 — AUTONOMOUS DEVELOPMENT LOOP
In each turn:
- Implement features, components, schemas, and routes incrementally.
- Run local validation after code changes (`npx tsc --noEmit`, `npm run build`).
- Update AC statuses (PENDING → IN_PROGRESS → DONE) as work is verified.
- Self-correction rule: if a command fails, read the exact error, fix the root
cause, and re-verify. Do not repeat the same failing command or edit more
than twice — change approach instead.
PHASE 3 — THE 4-GATE CONVERGENCE CHECK (MANDATORY)
Do not end execution or declare the project finished until all four gates pass
in the same turn:
Gate 1 — Compiler and build
`npx tsc --noEmit` and `npm run build` both exit 0 with zero errors.
Gate 2 — Dev server and runtime health
`npm run dev` boots cleanly with zero unhandled console or network errors.
Gate 3 — Acceptance criteria complete
Every item in `ac-matrix.json` has "status": "DONE".
Gate 4 — Design audit score >= 8.5/10
Audit visual hierarchy, color consistency, typography scale, spacing,
transitions, responsive behavior, and empty states. Score out of 10. If
below 8.5, refine and re-audit — but no more than 3 design passes total.
After 3 passes, stop and report the final score as-is.
================================================================================
3. FINAL COMPLETION OUTPUT
================================================================================
Only when all four gates pass, output:
- Final status: PROJECT COMPLETE & VERIFIED
- App summary and architecture overview
- Build and test commands executed, with exit codes
- Completed acceptance-criteria summary (including any BLOCKED items)
- Final design score (X/10) and UX highlights
- Instructions for running the app locally
Then stop.
预生产检查清单
在代理循环正式上线之前，需根据以下标准进行核查：
- 通过确定性测试或编译器以及外部退出码检查来执行目标验证。此项为必做项。
- 轮次限制，即固定的迭代次数上限，例如
currentTurn >= maxTurns。此项为必做项。 - 令牌预算，即累计使用上限，例如
totalTokens >= tokenLimit。此项为必做项。 - 超时限制，即基于实际时间的截止时间，例如
elapsed >= timeoutMs。此项为必做项。 - 进度保护机制，用于检测重复工具签名的滑动窗口检测器。建议使用。
- 状态保护机制，通过工作区差异哈希值来检测是否无变化。建议使用。
git stash实现回滚的断路器功能，为必需项。约束优于智能
可靠的自主智能体并非源于让模型变得更聪明，而更多取决于对系统运行界限的合理设计。只有四点要素才能区分一个脆弱的实验性系统与一个可以整个周末持续运行的稳定系统：不依赖模型的验证机制、严格的资源上限、进度哈希记录以及可恢复的回滚功能。