自愈型 Kubernetes：整合 OpenTelemetry、SigNoz 以及基于 Groq 的功能
《自愈型 Kubernetes 实战指南》：详细讲解如何整合 OpenTelemetry、SigNoz 以及基于 Groq 的功能，为采用该架构的团队提供合约定义、校验机制及可直接插入的代码模块。
可将此内容视为《自愈型 Kubernetes：集成 OpenTelemetry、SigNoz 以及基于 Groq 的修复代理（第 6 部分）》中理念面向操作人员的重构版本：清晰的阶段划分、有序的代码模块，以及能在交接过程中保留的恢复说明。 将“概览”阶段视为可度量的界面使用效果最佳。在扩大范围之前，先记录一份标准操作流程、一个故障案例以及回滚说明。 需同时记录正常运行路径和恢复路径。重试机制、人工审核环节以及死信处理都是产品本身的组成部分，而非后续需要补充的内容。
cd ~/k8s-aiops-blog
mkdir -p dashboard/backend dashboard/frontend
cd ~/k8s-aiops-blog/dashboard/backend
cat > main.py <<'EOF'
"""
Approval dashboard backend.
Receives incident cards from the remediation agent, serves them to the
React frontend, and applies approved kubectl commands via the Kubernetes
Python client.
"""
import logging
import shlex
import subprocess
import time
from typing import Literal
from uuid import uuid4from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModellogging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s — %(message)s",
)
log = logging.getLogger("dashboard")app = FastAPI(title="Approval Dashboard")app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_methods=["*"],
allow_headers=["*"],
)# ---------------------------------------------------------------------------
# In-memory store. Fine for a demo; swap for SQLite or Redis in production.
# ---------------------------------------------------------------------------
INCIDENTS: dict[str, dict] = {}# ---------------------------------------------------------------------------
# kubectl command allowlist.
# The LLM proposes a fix. We only execute it if it matches one of these
# safe prefixes. Nothing with delete, drain, or --all gets through.
# ---------------------------------------------------------------------------
ALLOWED_PREFIXES = (
"kubectl set env",
"kubectl rollout restart",
"kubectl scale",
"kubectl set resources",
)BLOCKED_TERMS = ("delete", "drain", "--all", "cluster-admin", "exec", "cp")
# ---------------------------------------------------------------------------
# Models
# ---------------------------------------------------------------------------
class IncidentCard(BaseModel):
id: str
timestamp: int
service: str
signals: dict
diagnosis: dict
status: Literal["pending", "approved", "rejected"] = "pending"
class ActionRequest(BaseModel):
actor: str = "operator"
# ---------------------------------------------------------------------------
# Routes
# ---------------------------------------------------------------------------
@app.get("/healthz")
def healthz():
return {"ok": True}
@app.post("/api/incidents", status_code=201)
def receive_incident(card: IncidentCard):
INCIDENTS[card.id] = card.dict()
log.info("incident received: id=%s type=%s", card.id, card.diagnosis.get("incident_type"))
return {"id": card.id}
@app.get("/api/incidents")
def list_incidents():
# Most recent first
return sorted(INCIDENTS.values(), key=lambda c: c["timestamp"], reverse=True)
@app.get("/api/incidents/{incident_id}")
def get_incident(incident_id: str):
if incident_id not in INCIDENTS:
raise HTTPException(status_code=404, detail="not found")
return INCIDENTS[incident_id]
@app.post("/api/incidents/{incident_id}/approve")
def approve_incident(incident_id: str, req: ActionRequest):
card = INCIDENTS.get(incident_id)
if not card:
raise HTTPException(status_code=404, detail="not found")
if card["status"] != "pending":
raise HTTPException(status_code=409, detail=f"incident already {card['status']}") proposed = card["diagnosis"].get("proposed_fix", "")
log.info("approve requested: id=%s fix=%r actor=%s", incident_id, proposed, req.actor) # Validate against allowlist
allowed = any(proposed.startswith(p) for p in ALLOWED_PREFIXES)
blocked = any(term in proposed for term in BLOCKED_TERMS) if not allowed or blocked:
log.error("REJECTED by allowlist: %r", proposed)
raise HTTPException(
status_code=400,
detail=f"proposed command did not pass allowlist validation: {proposed!r}",
) # Apply the fix
try:
args = shlex.split(proposed)
result = subprocess.run(
args,
capture_output=True,
text=True,
timeout=30,
)
if result.returncode != 0:
log.error("kubectl failed: %s", result.stderr)
raise HTTPException(status_code=500, detail=f"kubectl error: {result.stderr}") log.info("kubectl succeeded: %s", result.stdout.strip())
card["status"] = "approved"
card["applied_at"] = int(time.time())
card["applied_by"] = req.actor
card["kubectl_output"] = result.stdout.strip()
INCIDENTS[incident_id] = card
return {"status": "approved", "kubectl_output": result.stdout.strip()} except subprocess.TimeoutExpired:
raise HTTPException(status_code=504, detail="kubectl timed out")
@app.post("/api/incidents/{incident_id}/reject")
def reject_incident(incident_id: str, req: ActionRequest):
card = INCIDENTS.get(incident_id)
if not card:
raise HTTPException(status_code=404, detail="not found")
if card["status"] != "pending":
raise HTTPException(status_code=409, detail=f"incident already {card['status']}") card["status"] = "rejected"
card["rejected_at"] = int(time.time())
card["rejected_by"] = req.actor
INCIDENTS[incident_id] = card
log.info("incident rejected: id=%s actor=%s", incident_id, req.actor)
return {"status": "rejected"}
EOF
cat > requirements.txt <<'EOF'
fastapi==0.115.0
uvicorn[standard]==0.32.0
pydantic==2.9.2
httpx==0.27.2
EOF
cat > Dockerfile <<'EOF'
FROM python:3.11-slim
# Install kubectl so the backend can apply fixes
RUN apt-get update && apt-get install -y curl && \
curl -LO "https://dl.k8s.io/release/$(curl -Ls https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \
install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl && \
rm kubectl && \
apt-get cleanWORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY main.py .EXPOSE 9000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "9000"]
EOF
cd ~/k8s-aiops-blog/dashboard/frontend
cat > index.html <<'EOF'
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0"/>
<title>SRE Approval Dashboard</title>
<script src="https://cdnjs.cloudflare.com/ajax/libs/react/18.2.0/umd/react.production.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/react-dom/18.2.0/umd/react-dom.production.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/babel-standalone/7.23.2/babel.min.js"></script>
<link href="https://cdnjs.cloudflare.com/ajax/libs/tailwindcss/2.2.19/tailwind.min.css" rel="stylesheet"/>
</head>
<body class="bg-gray-950 text-gray-100 min-h-screen">
<div id="root"></div><script type="text/babel">
const API = "http://localhost:9000";const SEVERITY_COLOR = {
critical: "bg-red-600",
high: "bg-orange-500",
medium: "bg-yellow-500",
low: "bg-blue-500",
};const TYPE_LABEL = {
crash_loop: "Crash Loop",
oom_kill: "OOM Kill",
latency_spike: "Latency Spike",
token_spike: "Token Spike",
unknown: "Unknown",
};function ConfidenceBadge({ value }) {
const pct = Math.round((value || 0) * 100);
const color = pct >= 80 ? "text-green-400" : pct >= 50 ? "text-yellow-400" : "text-red-400";
return (
<span className={`text-xs font-mono font-bold ${color}`}>
{pct}% confidence
</span>
);
}function IncidentCard({ card, onAction }) {
const d = card.diagnosis || {};
const sev = d.severity || "low";
const isPending = card.status === "pending"; return (
<div className="bg-gray-900 border border-gray-700 rounded-xl p-5 mb-4 shadow-lg">
{/* Header row */}
<div className="flex items-center justify-between mb-3">
<div className="flex items-center gap-2">
<span className={`text-xs font-bold px-2 py-0.5 rounded-full text-white ${SEVERITY_COLOR[sev] || "bg-gray-600"}`}>
{sev.toUpperCase()}
</span>
<span className="font-semibold text-white">
{TYPE_LABEL[d.incident_type] || d.incident_type}
</span>
<span className="text-gray-400 text-sm">— {card.service}</span>
</div>
<div className="flex items-center gap-3">
<ConfidenceBadge value={d.confidence} />
<span className="text-xs text-gray-500">
{new Date(card.timestamp * 1000).toLocaleTimeString()}
</span>
</div>
</div> {/* Root cause */}
<p className="text-sm text-gray-300 mb-3">
<span className="text-gray-500 mr-1">Root cause:</span>
{d.root_cause}
</p> {/* Proposed fix */}
<div className="bg-gray-800 rounded-lg px-4 py-2 mb-4 font-mono text-sm text-green-300 flex items-center gap-2">
<span className="text-gray-500 select-none">lt;/span>
{d.proposed_fix}
</div> {/* Low confidence warning */}
{(d.confidence || 0) < 0.6 && isPending && (
<div className="bg-yellow-900 border border-yellow-600 rounded-lg px-3 py-2 mb-3 text-yellow-300 text-xs">
⚠ Low confidence — review signals carefully before approving.
</div>
)} {/* Action buttons or status badge */}
{isPending ? (
<div className="flex gap-3 mt-2">
<button
onClick={() => onAction(card.id, "approve")}
className="bg-green-600 hover:bg-green-500 text-white text-sm font-semibold px-5 py-2 rounded-lg transition"
>
✓ Approve
</button>
<button
onClick={() => onAction(card.id, "reject")}
className="bg-red-700 hover:bg-red-600 text-white text-sm font-semibold px-5 py-2 rounded-lg transition"
>
✗ Reject
</button>
</div>
) : (
<div className={`inline-block text-xs font-bold px-3 py-1 rounded-full mt-1 ${
card.status === "approved" ? "bg-green-800 text-green-300" : "bg-red-900 text-red-300"
}`}>
{card.status === "approved" ? "✓ Approved" : "✗ Rejected"}
{card.applied_by && ` by ${card.applied_by}`}
</div>
)} {/* kubectl output on approved cards */}
{card.kubectl_output && (
<div className="mt-3 bg-gray-800 rounded-lg px-4 py-2 font-mono text-xs text-gray-400">
{card.kubectl_output}
</div>
)}
</div>
);
}function App() {
const [incidents, setIncidents] = React.useState([]);
const [loading, setLoading] = React.useState(true);
const [error, setError] = React.useState(null);
const [acting, setActing] = React.useState(null); const fetchIncidents = () => {
fetch(`${API}/api/incidents`)
.then(r => r.json())
.then(data => { setIncidents(data); setLoading(false); setError(null); })
.catch(e => { setError(e.message); setLoading(false); });
}; React.useEffect(() => {
fetchIncidents();
const id = setInterval(fetchIncidents, 10000); // poll every 10s
return () => clearInterval(id);
}, []); const handleAction = async (id, action) => {
setActing(id);
try {
const r = await fetch(`${API}/api/incidents/${id}/${action}`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ actor: "operator" }),
});
if (!r.ok) {
const err = await r.json();
alert(`Action failed: ${err.detail}`);
}
fetchIncidents();
} catch (e) {
alert(`Network error: ${e.message}`);
} finally {
setActing(null);
}
}; const pending = incidents.filter(c => c.status === "pending");
const resolved = incidents.filter(c => c.status !== "pending"); return (
<div className="max-w-3xl mx-auto px-4 py-8">
{/* Header */}
<div className="mb-8">
<h1 className="text-2xl font-bold text-white">SRE Approval Dashboard</h1>
<p className="text-gray-400 text-sm mt-1">
Human-in-the-loop remediation — review every fix before it runs
</p>
</div> {loading && <p className="text-gray-500">Loading incidents…</p>}
{error && <p className="text-red-400">Could not reach backend: {error}</p>} {/* Pending */}
{pending.length > 0 && (
<section className="mb-8">
<h2 className="text-sm font-semibold text-gray-400 uppercase tracking-widest mb-3">
Awaiting Review ({pending.length})
</h2>
{pending.map(c => (
<IncidentCard key={c.id} card={c} onAction={handleAction} />
))}
</section>
)} {pending.length === 0 && !loading && (
<div className="bg-gray-900 border border-gray-700 rounded-xl p-8 text-center text-gray-500 mb-8">
No pending incidents. The cluster looks healthy.
</div>
)} {/* Resolved */}
{resolved.length > 0 && (
<section>
<h2 className="text-sm font-semibold text-gray-400 uppercase tracking-widest mb-3">
Resolved ({resolved.length})
</h2>
{resolved.map(c => (
<IncidentCard key={c.id} card={c} onAction={handleAction} />
))}
</section>
)}
</div>
);
}ReactDOM.createRoot(document.getElementById("root")).render(<App />);
</script>
</body>
</html>
EOF
cat > Dockerfile <<'EOF'
FROM nginx:alpine
COPY index.html /usr/share/nginx/html/index.html
EXPOSE 80
EOF
cd ~/k8s-aiops-blog/dashboard/backend
docker build -t approval-backend:v1 .
kind load docker-image approval-backend:v1 --name aiops
cd ~/k8s-aiops-blog/dashboard/frontend
docker build -t approval-frontend:v1 .
kind load docker-image approval-frontend:v1 --name aiops
cat > ~/k8s-aiops-blog/k8s/dashboard.yaml <<'EOF'
# ── Backend ────────────────────────────────────────────────────────────────
apiVersion: apps/v1
kind: Deployment
metadata:
name: approval-backend
namespace: apps
spec:
replicas: 1
selector:
matchLabels:
app: approval-backend
template:
metadata:
labels:
app: approval-backend
spec:
containers:
- name: backend
image: approval-backend:v1
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9000
readinessProbe:
httpGet:
path: /healthz
port: 9000
initialDelaySeconds: 5
periodSeconds: 10
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "256Mi"
cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
name: approval-dashboard
namespace: apps
spec:
selector:
app: approval-backend
ports:
- port: 9000
targetPort: 9000
---
# ── Frontend ───────────────────────────────────────────────────────────────
apiVersion: apps/v1
kind: Deployment
metadata:
name: approval-frontend
namespace: apps
spec:
replicas: 1
selector:
matchLabels:
app: approval-frontend
template:
metadata:
labels:
app: approval-frontend
spec:
containers:
- name: frontend
image: approval-frontend:v1
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
resources:
requests:
memory: "64Mi"
cpu: "50m"
limits:
memory: "128Mi"
cpu: "200m"
---
apiVersion: v1
kind: Service
metadata:
name: approval-frontend-svc
namespace: apps
spec:
selector:
app: approval-frontend
ports:
- port: 3000
targetPort: 80
EOF
kubectl apply -f ~/k8s-aiops-blog/k8s/dashboard.yaml
kubectl rollout status deployment/approval-backend -n apps
kubectl rollout status deployment/approval-frontend -n apps
kubectl port-forward -n apps svc/approval-dashboard 9000:9000
kubectl port-forward -n apps svc/approval-frontend-svc 3000:3000
kubectl logs -n apps deployment/remediation-agent --tail=5 -f
kubectl get pods -n apps -w
kubectl set env deployment/summarizer -n apps KILL_ME=true
curl -s http://localhost:8000/crash
... WARNING crash loop signal: error_rate=1.00
... INFO anomaly detected: crash_loop — calling Groq for diagnosis
... INFO diagnosis: type=crash_loop severity=critical confidence=0.91
fix=kubectl set env deployment/summarizer -n apps KILL_ME-
... INFO incident posted: id=<uuid> type=crash_loop
CRITICAL Crash Loop — summarizer 91% confidence
Root cause: The KILL_ME environment variable is set to true, causing the process to exit immediately on each start.
$ kubectl set env deployment/summarizer -n apps KILL_ME-
[ ✓ Approve ] [ ✗ Reject ]
✓ Approved by operator
deployment.apps/summarizer env updated
操作检查清单
在处理操作检查清单阶段时，首先写下相关合同条款：所需输入、成功信号以及部分失败时的处理方式。这样的检查清单能确保后续的代码修改保持一致性。
将配置信息与应用程序代码分开存放。环境文件、密钥存储以及功能开关应集中于一个位置，以便操作人员无需查看整个系统结构即可进行审计。
在成本较高的步骤之后设置检查点。当操作人员重新尝试后续节点时，恢复流程不应再次计费相同的LLM调用费用。
锁定依赖版本，并记录用于运行演示的图像摘要。可重复性远胜于经验主义。
同时记录正常流程和故障恢复流程。重试机制、人工审核环节以及错误处理都是产品不可或缺的部分，而非后续才需要补充的功能。
在成本较高的操作之后设置检查点。当操作员重新尝试后续节点时，恢复流程不应再次调用相同的大型语言模型接口。
在升级技术栈之前，先冻结版本，为关键流程保存标准化的操作记录，并确认回滚步骤。共享环境需要设置速率限制、租户验证机制，以及明确的密钥轮换负责人。与其追求花哨的一次性演示，不如注重扎实的可靠性。
针对4a22289d433c的批量处理说明：不要将提供商密钥放入代码仓库，为每个会话设置令牌使用上限，并将操作记录与评估用文件存放在同一位置，以便后续更换模型时仍能保持数据可比性。
在实施强化措施的第0阶段，应在修改代码之前明确输入参数、该步骤的负责人以及结束标准。操作人员应能够从已知的检查点重新运行该步骤，而无需猜测隐藏状态。 配置信息应置于应用程序代码之外。环境文件、密钥存储以及功能标志应集中存放于一个位置，这样操作人员无需查看整个系统结构即可进行审计。
强化措施细节0/765：针对此措施需统计耗时、错误类型以及令牌使用情况，然后依据固定的评估标准而非主观判断来决定是否保留该变更。
在处理强化措施的第一阶段时，首先写下相关约定：所需的输入参数、成功信号以及部分失败时的处理方式。这样的检查清单能确保后续的代码修改保持一致性。 相比冗长的脚本，更应采用小型且可测试的单元。当某个步骤失败时，故障应能指向单一责任点，而非复杂的流程链。
强化措施细节 1/765：需测量该步骤的运行时间、错误类型以及代币消耗情况，然后依据固定的评估标准而非主观感受来决定是否保留该修改。
将强化措施的第二阶段视为可度量的对象来处理效果最佳。在扩大范围之前，先记录一份理想的运行示例、一个失败案例以及回滚说明。 在功能结果旁同时记录时间消耗及代币或查询成本。提前了解成本情况，可避免在从演示环境过渡到共享环境时出现意外支出。
强化措施细节 2/765：记录该任务的执行时间、错误类型以及令牌消耗情况，然后依据固定的问题清单而非个人经验来判断是否保留该变更。
在强化措施的第3阶段，应在修改代码之前明确输入参数、该步骤的负责人以及完成标准。操作人员应能够从已知的检查点重新运行该步骤，而无需猜测隐藏状态。需同时记录正常流程和异常恢复流程。重试机制、人工审核环节以及错误处理方式都是产品本身的组成部分，而非后续的优化工作。
强化措施细节 3/765：记录该任务的执行时间、错误类型以及令牌消耗情况，然后依据固定的问题清单而非个人经验来判断是否保留该变更。