Self-Healing Kubernetes in Action: Three Failures, SigNoz Signals and Approved Fixes

The demo has several moving parts. A local Kind cluster hosts SigNoz for traces and logs, plus a Python app whose three endpoints are broken on purpose. A remediation agent polls for anomalies and asks Groq for a diagnosis, and a React dashboard lets an operator approve or reject the suggested fix before anything runs. Once all of that is built, the interesting part is watching it behave. This run-through triggers each failure with the full stack live, looks at what SigNoz records for each incident and lists what must change before anyone trusts the setup beyond a laptop.

Confirm the stack is up

Five things should be running before any failure is triggered. The first check is that every pod is healthy in both namespaces.

# All pods should show Running
kubectl get pods -n observability
kubectl get pods -n apps

The port-forwards for SigNoz, the summarizer service and the dashboard each need their own terminal tab, and the last command in the block tails the agent logs.

# Port-forwards (run each in its own tab if not already open)
# Tab A: kubectl port-forward -n observability svc/signoz 8080:8080
# Tab B: kubectl port-forward -n apps svc/summarizer 8000:80
# Tab C: kubectl port-forward -n apps svc/approval-dashboard 9000:9000
# Tab D: kubectl port-forward -n apps svc/approval-frontend-svc 3000:3000# Confirm the agent is polling
kubectl logs -n apps deployment/remediation-agent --tail=5

Recent all-clear lines in that output mean the agent is healthy. A pod that is not Running should be inspected with kubectl describe and repaired first. Keep two browser tabs visible: SigNoz on port 8080 and the dashboard on port 3000.

Scenario one: CrashLoopBackOff

Setting a KILL_ME environment variable on the summarizer deployment and then hitting its crash endpoint makes the process die mid-request.

kubectl set env deployment/summarizer -n apps KILL_ME=true
curl -s http://localhost:8000/crash

The curl call may return nothing or hang, since the process is exiting while it runs. A watch on the pods shows the restart counter climbing and the status flipping between Running and CrashLoopBackOff, and two or three cycles are enough.

kubectl get pods -n apps -w

In SigNoz. The summarizer service overview shows the error rate jumping to 100% as soon as the pod starts crashing, while the request rate falls to zero whenever the pod is down between restarts. Traffic, then nothing, then traffic again is the visual signature of a crash loop. A deployment rollout looks different, with a clean cutover and no error spike, and a network partition looks different too, with the error rate rising while the pod stays Running.

The Logs tab, filtered to the summarizer, contains the critical line the app emitted just before it died.

CRITICAL summarizer — KILL_ME is true, exiting process

That log carries the trace ID of the request that triggered it, so one click leads from the log line to the trace and to the exact HTTP request that ended the process.

The agent reacts. Within 30 seconds of the first crash the polling cycle fires. It sees an error rate of 1.0 against a threshold of 0.10, bundles the signal and calls Groq. The agent's logs show the sequence.

kubectl logs -n apps deployment/remediation-agent --tail=20 -f

WARNING  agent — crash loop signal: error_rate=1.00
INFO     agent — anomaly detected: crash_loop — calling Groq for diagnosis
INFO     agent — diagnosis: type=crash_loop severity=critical confidence=0.91
           fix=kubectl set env deployment/summarizer -n apps KILL_ME-
INFO     incident_poster — incident posted: id=<uuid> type=crash_loop

Approval. The dashboard shows the card within about ten seconds, and the proposed fix is the right one, so the operator clicks Approve. The backend checks that the command begins with kubectl set env and contains no blocked terms, and then runs it. The card updates to confirm the approval and the environment change.

✓ Approved by operator
deployment.apps/summarizer env updated

Back in the pod watch, the next start succeeds, the crash loop ends and the SigNoz error rate returns to zero. Before the next scenario, a quick check confirms the variable is gone.

# The approve action already removed KILL_ME, but confirm:
kubectl get deployment summarizer -n apps -o jsonpath='{.spec.template.spec.containers[0].env}' | jq

KILL_ME should not appear in the output.

Scenario two: OOMKill

A loop of POST requests to the leak endpoint grows a bucket in memory a little at a time.

for i in {1..30}; do
  curl -s -X POST http://localhost:8000/leak | jq -c '{leaked_mb,chunks}'
  sleep 0.5
done

Memory climbs in the pod watch, and around the 25th call the pod is killed mid-loop, after which curl starts returning connection errors.

In SigNoz. This time the error rate does not rise cleanly. An OOM kill is abrupt, and the application never returns an error response. The request rate instead falls off a cliff: traffic, a hard stop, a gap, then traffic again once the pod restarts.

The logs hold the real story. Filtered to the summarizer, the gap is preceded by warnings with a growing bucket size.

WARNING summarizer — leak endpoint hit, bucket now holds 10 MB
WARNING summarizer — leak endpoint hit, bucket now holds 20 MB
...
WARNING summarizer — leak endpoint hit, bucket now holds 240 MB

After the last warning comes silence, with no exception, no stack trace and no graceful shutdown message, because the kernel's OOM killer gives the process no chance to log. A run of climbing warnings followed by nothing is the pattern worth recognizing.

The cluster confirms the cause from its side.

kubectl describe pod -n apps -l app=summarizer | grep -A 5 "Last State"

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

Exit code 137 is the kernel's signature, SIGKILL (signal 9) plus 128. It separates an OOM kill from an application crash, which exits with code 1, and from a liveness probe failure, which also gives 137 but a different reason string in the describe output.

The agent reacts. Its OOM check searches the SigNoz log index for the string OOMKilled over the last ten minutes. A hit makes it bundle log samples and the pod restart event into a prompt, and the diagnosis comes back.

type=oom_kill severity=critical confidence=0.88
fix=kubectl set resources deployment/summarizer -n apps \
  --containers=summarizer --limits=memory=512Mi

The suggested fix doubles the memory limit. As a short-term measure that is reasonable, since it buys time to ship the application fix. After Approve, the backend applies a 512Mi limit, Kubernetes performs a rolling restart and the pod returns with headroom.

The leak itself remains in the code. The change only raises the ceiling, and a real incident would end with a code fix that clears the leak bucket periodically. The agent's job is to stop the bleeding, and the root cause is still a human task.

Scenario three: latency and token spike

A short loop of calls to the expensive endpoint produces the third failure.

for i in {1..5}; do
  curl -s -X POST http://localhost:8000/expensive \
    | jq '{output_tokens, length_chars}'
  sleep 2
done

Nothing crashes and nothing is killed, and the pod stays Running. Responses take 8 to 15 seconds each, though, and token counts sit near the 2,048 max_tokens ceiling.

In SigNoz. This quiet failure is where the observability stack earns its place. The p95 latency line, flat near 1.5 seconds during normal summarize traffic, jumps when the expensive calls land, and the shape is a step change and not a slow drift, since every call hits the ceiling. Sorting traces by duration puts those calls at the top at 8 to 15 seconds each.

Opening a slow trace in the flame graph shows nearly all the time inside a single child span, the outbound HTTP request to api.groq.com. The span for the endpoint handler itself is thin, because the FastAPI handler does almost nothing, and the LLM round trip dominates. The span's response content length is far larger than for a normal summarize call, which is the output token count showing up as bytes on the wire. Flat line, step up, flat again when the loop ends: that is the profile of a misconfigured endpoint, not an infrastructure fault.

The agent reacts. The latency check compares p95 by route. The expensive route comes back around 11,000 ms against a 5,000 ms threshold, and the agent bundles the breakdown.

{
  "check": "latency_spike",
  "spikes": { "/expensive": 11240.0 },
  "p95_ms_by_route": {
    "/summarize": 1820.0,
    "/expensive": 11240.0,
    "/healthz": 4.0
  }
}

Groq reads it and produces a precise diagnosis.

type=token_spike severity=high confidence=0.85
root_cause: The /expensive endpoint has no reasonable max_tokens budget
            and a prompt that invites long outputs, causing p95 latency
            to spike to 11s versus 1.8s on the normal /summarize path.
fix=kubectl set env deployment/summarizer -n apps MAX_TOKENS_EXPENSIVE=256

The proposed fix sets an environment variable the app can read to cap the token budget on that endpoint. After Approve, the variable is set. A real deployment would also change the app to read MAX_TOKENS_EXPENSIVE from the environment. The agent proposes a configuration lever and not a code change, which is the right scope for a kubectl-level fix.

Three failures on one timeline

With all three scenarios run in sequence, the summarizer overview over a two-hour window shows three distinct signatures.

• The crash loop is a saw-tooth in the error rate that spikes to 100%, drops to zero and spikes again, with matching gaps in throughput.
• The OOM kill is a single hard cliff in request rate with no error spike, because the process never got to return an error.
• The token spike is a clean step in p95 latency while the error rate stays flat, because the pod never failed and only became slow and expensive.

Three failure modes with three recognizable shapes appear on one dashboard, with no custom alert rules and no manual log parsing. That is the payoff of correlated traces, metrics and logs.

Changes needed for production

The demo runs on a laptop and cuts corners that matter elsewhere.

Replace the in-memory incident store. The incident list is a Python dict inside the backend pod, so a restart loses every card. SQLite through SQLAlchemy is a one-line change that needs no extra service, and Redis fits when the dashboard must scale across several backend replicas. The API shape stays the same and only the storage behind it changes.

Add Slack notifications. As it stands, someone has to be watching the dashboard to know a card appeared. A Slack webhook call inside the backend's incident receiver fixes that.

import httpx

SLACK_WEBHOOK = os.environ.get("SLACK_WEBHOOK_URL")def notify_slack(card: dict):
    if not SLACK_WEBHOOK:
        return
    d = card["diagnosis"]
    httpx.post(SLACK_WEBHOOK, json={
        "text": (
            f"*{d['severity'].upper()} — {d['incident_type']}* on `{card['service']}`\n"
            f"Root cause: {d['root_cause']}\n"
            f"Proposed fix: `{d['proposed_fix']}`\n"
            f"Confidence: {int(d['confidence']*100)}%\n"
            f"<http://your-dashboard-url|Review on dashboard>"
        )
    })

Calling the notifier at the end of the receiver means each new incident pings the on-call channel, and the approver no longer needs to watch a browser tab.

Handle cluster credentials properly. The backend shells out to kubectl, which is fine when the node environment configures it. In a real cluster the backend pod needs a ServiceAccount with suitable RBAC permissions and the Kubernetes Python client set up for in-cluster credentials.

from kubernetes import client, config

config.load_incluster_config()
apps_v1 = client.AppsV1Api()
core_v1 = client.CoreV1Api()

Each approved command type then becomes a direct SDK call and not a shell string. That is more verbose, but it gives structured error handling, audit logging at the API server and no shell injection risk.

Tighten the allowlist. The demo accepts any kubectl set env command. Production should restrict it to named deployments and namespaces.

ALLOWED_TARGETS = {
    "deployment/summarizer -n apps",
    "deployment/worker -n apps",
}

def validate_command(cmd: str) -> bool:
    if not any(cmd.startswith(p) for p in ALLOWED_PREFIXES):
        return False
    if any(term in cmd for term in BLOCKED_TERMS):
        return False
    if not any(target in cmd for target in ALLOWED_TARGETS):
        return False
    return True

A model must not be able to aim a fix at any namespace or deployment outside the list it was given.

Projects worth studying next

The demo was built from scratch to show how the pieces connect, and production use would build on existing projects.

• KubeStellar helps when workloads span several clusters. It absorbs the control-plane work that complicates remediation across clusters, for example identifying which cluster an incident lives in before any fix is suggested.
• OpenTelemetry goes much deeper than the demo needed. The Collector has processors and exporters that can add Kubernetes metadata to spans automatically, so resource attributes need not be added by hand in app manifests.
• KEDA covers the class of scale-this-deployment fixes the agent sometimes proposes and a human should not approve one at a time. If the fix is always to scale up when queue depth passes some number, that belongs in a KEDA ScaledObject and not in a human approval.
• Robusta and HolmesGPT are production-grade relatives of what was built here. After seeing the query layer, the prompt structure and the approval gate, reading their code becomes easy to follow.

What the whole build contains

The finished system is a three-node Kind cluster with a realistic multi-node scheduling topology, and SigNoz deployed through Helm that ingests traces, metrics and logs via the OpenTelemetry Collector. A Python FastAPI service is auto-instrumented at startup, so every HTTP request and LLM call becomes a trace and every log line is correlated without instrumentation code in the app.

Three broken endpoints trigger the three most common Kubernetes failures: a process crash, memory exhaustion and a cost and latency regression. A remediation agent polls SigNoz every 30 seconds, applies threshold checks, bundles the signals into a structured prompt, asks an LLM for a diagnosis and posts an incident card. A FastAPI backend receives the cards, validates proposed fixes against an allowlist and runs approved commands through kubectl. The React dashboard adds approval and rejection controls, a badge showing model confidence, a banner for shaky diagnoses and a log of each action taken.

Everything runs locally on free-tier APIs and costs nothing. The code is small enough, around 600 lines of Python with one HTML page and a few YAML manifests, that every piece can be understood end to end.