Kubernetes修复代理的审批控制面板:FastAPI与React实现
构建一个基于 FastAPI 的后端系统,结合 kubectl 允许列表与 React 批准控制面板,使得大型语言模型修复代理只能在人类用户点击“批准”后才能处理故障。
来源: Self-Healing Kubernetes: Wiring OpenTelemetry, SigNoz, and a Groq-Powered Remediation Agent (PART 6)
上一阶段的修复代理已经能够识别三种类型的事故,并向Groq请求诊断。随后它会尝试将事故卡片发送到尚未存在的后端。这一阶段会构建该后端以及其前的React控制面板,从而形成闭环。当发生事故时,SigNoz会记录下来,代理进行诊断,控制面板上会出现相应卡片,操作员点击“批准”后,修复措施就会在集群中执行。完成初始设置后便无需终端。
各组件之间的关联方式
修复工具通过 POST /api/incidents 将事件卡片发送到后端。FastAPI 服务负责存储这些卡片,验证 kubectl 命令,并通过 Kubernetes 工具应用已批准的修复方案。React 控制面板通过 GET /api/incidents 获取卡片列表,再通过 POST /api/incidents/{id}/approve 或 /reject 对相应卡片执行操作。界面上会显示事件卡片、批准与拒绝按钮、置信度标识以及审计日志。
集群中运行着两个容器:React 应用是由 nginx 提供的静态构建版本,而 FastAPI 后端是唯一允许访问 Kubernetes API 的组件。
步骤 1:创建文件夹
这两个部分都位于项目中的 dashboard 目录下:
cd ~/k8s-aiops-blog
mkdir -p dashboard/backend dashboard/frontend
步骤2:FastAPI后端
后端承担四项功能:将传入的卡片信息存储在内存中,由于没有涉及数据库,这对演示来说已经足够;将这些卡片信息提供给前端;处理“批准”和“拒绝”操作。在接收到“批准”指令时,它会验证相应的kubectl命令,然后执行该命令。请从后端文件夹开始操作:
cd ~/k8s-aiops-blog/dashboard/backend
然后创建 main.py:
cat > main.py <<'EOF'
"""
Approval dashboard backend.
Receives incident cards from the remediation agent, serves them to the
React frontend, and applies approved kubectl commands via the Kubernetes
Python client.
"""
import logging
import shlex
import subprocess
import time
from typing import Literal
from uuid import uuid4from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModellogging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s — %(message)s",
)
log = logging.getLogger("dashboard")app = FastAPI(title="Approval Dashboard")app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_methods=["*"],
allow_headers=["*"],
)# ---------------------------------------------------------------------------
# In-memory store. Fine for a demo; swap for SQLite or Redis in production.
# ---------------------------------------------------------------------------
INCIDENTS: dict[str, dict] = {}# ---------------------------------------------------------------------------
# kubectl command allowlist.
# The LLM proposes a fix. We only execute it if it matches one of these
# safe prefixes. Nothing with delete, drain, or --all gets through.
# ---------------------------------------------------------------------------
ALLOWED_PREFIXES = (
"kubectl set env",
"kubectl rollout restart",
"kubectl scale",
"kubectl set resources",
)BLOCKED_TERMS = ("delete", "drain", "--all", "cluster-admin", "exec", "cp")
# ---------------------------------------------------------------------------
# Models
# ---------------------------------------------------------------------------
class IncidentCard(BaseModel):
id: str
timestamp: int
service: str
signals: dict
diagnosis: dict
status: Literal["pending", "approved", "rejected"] = "pending"
class ActionRequest(BaseModel):
actor: str = "operator"
# ---------------------------------------------------------------------------
# Routes
# ---------------------------------------------------------------------------
@app.get("/healthz")
def healthz():
return {"ok": True}
@app.post("/api/incidents", status_code=201)
def receive_incident(card: IncidentCard):
INCIDENTS[card.id] = card.dict()
log.info("incident received: id=%s type=%s", card.id, card.diagnosis.get("incident_type"))
return {"id": card.id}
@app.get("/api/incidents")
def list_incidents():
# Most recent first
return sorted(INCIDENTS.values(), key=lambda c: c["timestamp"], reverse=True)
@app.get("/api/incidents/{incident_id}")
def get_incident(incident_id: str):
if incident_id not in INCIDENTS:
raise HTTPException(status_code=404, detail="not found")
return INCIDENTS[incident_id]
@app.post("/api/incidents/{incident_id}/approve")
def approve_incident(incident_id: str, req: ActionRequest):
card = INCIDENTS.get(incident_id)
if not card:
raise HTTPException(status_code=404, detail="not found")
if card["status"] != "pending":
raise HTTPException(status_code=409, detail=f"incident already {card['status']}") proposed = card["diagnosis"].get("proposed_fix", "")
log.info("approve requested: id=%s fix=%r actor=%s", incident_id, proposed, req.actor) # Validate against allowlist
allowed = any(proposed.startswith(p) for p in ALLOWED_PREFIXES)
blocked = any(term in proposed for term in BLOCKED_TERMS) if not allowed or blocked:
log.error("REJECTED by allowlist: %r", proposed)
raise HTTPException(
status_code=400,
detail=f"proposed command did not pass allowlist validation: {proposed!r}",
) # Apply the fix
try:
args = shlex.split(proposed)
result = subprocess.run(
args,
capture_output=True,
text=True,
timeout=30,
)
if result.returncode != 0:
log.error("kubectl failed: %s", result.stderr)
raise HTTPException(status_code=500, detail=f"kubectl error: {result.stderr}") log.info("kubectl succeeded: %s", result.stdout.strip())
card["status"] = "approved"
card["applied_at"] = int(time.time())
card["applied_by"] = req.actor
card["kubectl_output"] = result.stdout.strip()
INCIDENTS[incident_id] = card
return {"status": "approved", "kubectl_output": result.stdout.strip()} except subprocess.TimeoutExpired:
raise HTTPException(status_code=504, detail="kubectl timed out")
@app.post("/api/incidents/{incident_id}/reject")
def reject_incident(incident_id: str, req: ActionRequest):
card = INCIDENTS.get(incident_id)
if not card:
raise HTTPException(status_code=404, detail="not found")
if card["status"] != "pending":
raise HTTPException(status_code=409, detail=f"incident already {card['status']}") card["status"] = "rejected"
card["rejected_at"] = int(time.time())
card["rejected_by"] = req.actor
INCIDENTS[incident_id] = card
log.info("incident rejected: id=%s actor=%s", incident_id, req.actor)
return {"status": "rejected"}
EOF
允许列表检查值得仔细研究。ALLOWED_PREFIXES用于确保命令以经过批准的安全操作开头。BLOCKED_TERMS是另一层过滤机制,无论命令前缀如何,只要包含不应被执行的词汇就会遭到拒绝。两者都必须通过验证,因此以kubectl set env开头但带有--all参数的命令仍会被拒绝。
为简化操作并便于审计,该代码使用了subprocess.run而非 Kubernetes Python SDK,这样就能轻松记录实际执行的命令内容。在生产环境中,人们更倾向于使用 SDK 的CoreV1Api或AppsV1Api,因为它们无需启动子进程,且能返回结构化的错误信息。
步骤 3:锁定依赖项
具体版本信息会写入requirements.txt文件中:
cat > requirements.txt <<'EOF'
fastapi==0.115.0
uvicorn[standard]==0.32.0
pydantic==2.9.2
httpx==0.27.2
EOF
第4步:后端Dockerfile
它以一个精简版的Python镜像为基础:
cat > Dockerfile <<'EOF'
FROM python:3.11-slim
其余步骤会安装 kubectl,以便后端能够应用修复,随后还会设置应用程序:
# Install kubectl so the backend can apply fixes
RUN apt-get update && apt-get install -y curl && \
curl -LO "https://dl.k8s.io/release/$(curl -Ls https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \
install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl && \
rm kubectl && \
apt-get cleanWORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY main.py .EXPOSE 9000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "9000"]
EOF
第5步:React前端
前端只是一个普通的HTML文件,它会从CDN加载React和Tailwind,因此无需编译,Dockerfile也相当简单。首先切换到前端文件夹:
cd ~/k8s-aiops-blog/dashboard/frontend
然后编写 index.html:
cat > index.html <<'EOF'
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0"/>
<title>SRE Approval Dashboard</title>
<script src="https://cdnjs.cloudflare.com/ajax/libs/react/18.2.0/umd/react.production.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/react-dom/18.2.0/umd/react-dom.production.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/babel-standalone/7.23.2/babel.min.js"></script>
<link href="https://cdnjs.cloudflare.com/ajax/libs/tailwindcss/2.2.19/tailwind.min.css" rel="stylesheet"/>
</head>
<body class="bg-gray-950 text-gray-100 min-h-screen">
<div id="root"></div><script type="text/babel">
const API = "http://localhost:9000";const SEVERITY_COLOR = {
critical: "bg-red-600",
high: "bg-orange-500",
medium: "bg-yellow-500",
low: "bg-blue-500",
};const TYPE_LABEL = {
crash_loop: "Crash Loop",
oom_kill: "OOM Kill",
latency_spike: "Latency Spike",
token_spike: "Token Spike",
unknown: "Unknown",
};function ConfidenceBadge({ value }) {
const pct = Math.round((value || 0) * 100);
const color = pct >= 80 ? "text-green-400" : pct >= 50 ? "text-yellow-400" : "text-red-400";
return (
<span className={`text-xs font-mono font-bold ${color}`}>
{pct}% confidence
</span>
);
}function IncidentCard({ card, onAction }) {
const d = card.diagnosis || {};
const sev = d.severity || "low";
const isPending = card.status === "pending"; return (
<div className="bg-gray-900 border border-gray-700 rounded-xl p-5 mb-4 shadow-lg">
{/* Header row */}
<div className="flex items-center justify-between mb-3">
<div className="flex items-center gap-2">
<span className={`text-xs font-bold px-2 py-0.5 rounded-full text-white ${SEVERITY_COLOR[sev] || "bg-gray-600"}`}>
{sev.toUpperCase()}
</span>
<span className="font-semibold text-white">
{TYPE_LABEL[d.incident_type] || d.incident_type}
</span>
<span className="text-gray-400 text-sm">— {card.service}</span>
</div>
<div className="flex items-center gap-3">
<ConfidenceBadge value={d.confidence} />
<span className="text-xs text-gray-500">
{new Date(card.timestamp * 1000).toLocaleTimeString()}
</span>
</div>
</div> {/* Root cause */}
<p className="text-sm text-gray-300 mb-3">
<span className="text-gray-500 mr-1">Root cause:</span>
{d.root_cause}
</p> {/* Proposed fix */}
<div className="bg-gray-800 rounded-lg px-4 py-2 mb-4 font-mono text-sm text-green-300 flex items-center gap-2">
<span className="text-gray-500 select-none">lt;/span>
{d.proposed_fix}
</div> {/* Low confidence warning */}
{(d.confidence || 0) < 0.6 && isPending && (
<div className="bg-yellow-900 border border-yellow-600 rounded-lg px-3 py-2 mb-3 text-yellow-300 text-xs">
⚠ Low confidence — review signals carefully before approving.
</div>
)} {/* Action buttons or status badge */}
{isPending ? (
<div className="flex gap-3 mt-2">
<button
onClick={() => onAction(card.id, "approve")}
className="bg-green-600 hover:bg-green-500 text-white text-sm font-semibold px-5 py-2 rounded-lg transition"
>
✓ Approve
</button>
<button
onClick={() => onAction(card.id, "reject")}
className="bg-red-700 hover:bg-red-600 text-white text-sm font-semibold px-5 py-2 rounded-lg transition"
>
✗ Reject
</button>
</div>
) : (
<div className={`inline-block text-xs font-bold px-3 py-1 rounded-full mt-1 ${
card.status === "approved" ? "bg-green-800 text-green-300" : "bg-red-900 text-red-300"
}`}>
{card.status === "approved" ? "✓ Approved" : "✗ Rejected"}
{card.applied_by && ` by ${card.applied_by}`}
</div>
)} {/* kubectl output on approved cards */}
{card.kubectl_output && (
<div className="mt-3 bg-gray-800 rounded-lg px-4 py-2 font-mono text-xs text-gray-400">
{card.kubectl_output}
</div>
)}
</div>
);
}function App() {
const [incidents, setIncidents] = React.useState([]);
const [loading, setLoading] = React.useState(true);
const [error, setError] = React.useState(null);
const [acting, setActing] = React.useState(null); const fetchIncidents = () => {
fetch(`${API}/api/incidents`)
.then(r => r.json())
.then(data => { setIncidents(data); setLoading(false); setError(null); })
.catch(e => { setError(e.message); setLoading(false); });
}; React.useEffect(() => {
fetchIncidents();
const id = setInterval(fetchIncidents, 10000); // poll every 10s
return () => clearInterval(id);
}, []); const handleAction = async (id, action) => {
setActing(id);
try {
const r = await fetch(`${API}/api/incidents/${id}/${action}`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ actor: "operator" }),
});
if (!r.ok) {
const err = await r.json();
alert(`Action failed: ${err.detail}`);
}
fetchIncidents();
} catch (e) {
alert(`Network error: ${e.message}`);
} finally {
setActing(null);
}
}; const pending = incidents.filter(c => c.status === "pending");
const resolved = incidents.filter(c => c.status !== "pending"); return (
<div className="max-w-3xl mx-auto px-4 py-8">
{/* Header */}
<div className="mb-8">
<h1 className="text-2xl font-bold text-white">SRE Approval Dashboard</h1>
<p className="text-gray-400 text-sm mt-1">
Human-in-the-loop remediation — review every fix before it runs
</p>
</div> {loading && <p className="text-gray-500">Loading incidents…</p>}
{error && <p className="text-red-400">Could not reach backend: {error}</p>} {/* Pending */}
{pending.length > 0 && (
<section className="mb-8">
<h2 className="text-sm font-semibold text-gray-400 uppercase tracking-widest mb-3">
Awaiting Review ({pending.length})
</h2>
{pending.map(c => (
<IncidentCard key={c.id} card={c} onAction={handleAction} />
))}
</section>
)} {pending.length === 0 && !loading && (
<div className="bg-gray-900 border border-gray-700 rounded-xl p-8 text-center text-gray-500 mb-8">
No pending incidents. The cluster looks healthy.
</div>
)} {/* Resolved */}
{resolved.length > 0 && (
<section>
<h2 className="text-sm font-semibold text-gray-400 uppercase tracking-widest mb-3">
Resolved ({resolved.length})
</h2>
{resolved.map(c => (
<IncidentCard key={c.id} card={c} onAction={handleAction} />
))}
</section>
)}
</div>
);
}ReactDOM.createRoot(document.getElementById("root")).render(<App />);
</script>
</body>
</html>
EOF
第6步:前端Dockerfile
使用nginx镜像来提供静态页面:
cat > Dockerfile <<'EOF'
FROM nginx:alpine
COPY index.html /usr/share/nginx/html/index.html
EXPOSE 80
EOF
第7步:生成图像并将其加载到Kind中
首先从后端开始,逐个生成图像并将其加载到本地集群中:
cd ~/k8s-aiops-blog/dashboard/backend
docker build -t approval-backend:v1 .
kind load docker-image approval-backend:v1 --name aiops
然后是前端部分:
cd ~/k8s-aiops-blog/dashboard/frontend
docker build -t approval-frontend:v1 .
kind load docker-image approval-frontend:v1 --name aiops
第8步:Kubernetes清单文件
一个dashboard.yaml文件中包含了两部分的部署任务和服务配置:
cat > ~/k8s-aiops-blog/k8s/dashboard.yaml <<'EOF'
# ── Backend ────────────────────────────────────────────────────────────────
apiVersion: apps/v1
kind: Deployment
metadata:
name: approval-backend
namespace: apps
spec:
replicas: 1
selector:
matchLabels:
app: approval-backend
template:
metadata:
labels:
app: approval-backend
spec:
containers:
- name: backend
image: approval-backend:v1
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9000
readinessProbe:
httpGet:
path: /healthz
port: 9000
initialDelaySeconds: 5
periodSeconds: 10
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "256Mi"
cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
name: approval-dashboard
namespace: apps
spec:
selector:
app: approval-backend
ports:
- port: 9000
targetPort: 9000
---
# ── Frontend ───────────────────────────────────────────────────────────────
apiVersion: apps/v1
kind: Deployment
metadata:
name: approval-frontend
namespace: apps
spec:
replicas: 1
selector:
matchLabels:
app: approval-frontend
template:
metadata:
labels:
app: approval-frontend
spec:
containers:
- name: frontend
image: approval-frontend:v1
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
resources:
requests:
memory: "64Mi"
cpu: "50m"
limits:
memory: "128Mi"
cpu: "200m"
---
apiVersion: v1
kind: Service
metadata:
name: approval-frontend-svc
namespace: apps
spec:
selector:
app: approval-frontend
ports:
- port: 3000
targetPort: 80
EOF
应用它,然后等待两次发布都完成:
kubectl apply -f ~/k8s-aiops-blog/k8s/dashboard.yaml
kubectl rollout status deployment/approval-backend -n apps
kubectl rollout status deployment/approval-frontend -n apps
第9步:打开控制面板
还需要进行两次端口转发,每一步都在一个新的终端标签页中操作。第一次转发用于暴露后端服务:
kubectl port-forward -n apps svc/approval-dashboard 9000:9000
第二个展示了前端界面:
kubectl port-forward -n apps svc/approval-frontend-svc 3000:3000
访问 http://localhost:3000,页面显示“暂无待处理事件”的空状态,目前这是正确的。
第10步:运行整个循环
所有内容都在这里交汇,因此请并排使用四个终端标签页。第一个标签页用于查看代理日志:
kubectl logs -n apps deployment/remediation-agent --tail=5 -f
第二个进程负责监控摘要服务中的任务组:
kubectl get pods -n apps -w
第三个会引发崩溃循环:
kubectl set env deployment/summarizer -n apps KILL_ME=true
curl -s http://localhost:8000/crash
第四个是位于 http://localhost:3000 的浏览器。
大约三十秒后,该代理会记录下崩溃循环信号,标记该异常,调用 Groq 并记录诊断结果及建议的修复方案:
... WARNING crash loop signal: error_rate=1.00
... INFO anomaly detected: crash_loop — calling Groq for diagnosis
... INFO diagnosis: type=crash_loop severity=critical confidence=0.91
fix=kubectl set env deployment/summarizer -n apps KILL_ME-
... INFO incident posted: id=<uuid> type=crash_loop
控制面板每十秒自动刷新一次,此时会显示事件卡片:
CRITICAL Crash Loop — summarizer 91% confidence
Root cause: The KILL_ME environment variable is set to true, causing the process to exit immediately on each start.
$ kubectl set env deployment/summarizer -n apps KILL_ME-
[ ✓ Approve ] [ ✗ Reject ]
点击“批准”后,后端会将该命令与允许列表进行比对,执行该命令并更新卡片信息:
✓ Approved by operator
deployment.apps/summarizer env updated
在容器监控标签页中,摘要生成器能够正常重启,并顺利进入“运行中 1/1”的状态,而不会陷入崩溃循环。循环已关闭。
保持允许列表的严谨性
该列表被刻意设置得较为保守。实际环境中可以根据团队已遵循的操作手册对其进行扩展。如果处理内存泄漏的标准方法始终是kubectl rollout restart,那就将其加入列表;如果有时需要使用kubectl scale,也应一并添加。
切勿添加任何可能造成破坏的操作。大型语言模型很少能始终做出正确判断,因此不能将其交给它来执行kubectl delete这类操作。设置审批步骤是因为有9%的情况是模型以91%的置信度给出错误判断,这类情况往往会在凌晨3点把人吵醒。
还剩下什么
接下来是几个扩展功能。若能完整演示崩溃循环、内存不足导致的进程终止以及令牌数量激增的情况,就能看到 SigNoz 为每种情况生成的时序图。内存中的事件存储应逐步被真正的数据库取代,Slack 可以用来通知每张新生成的卡片,而 CNCF 生态系统中也有许多值得研究、可用于实际生产的环境。