AegisOps: An Autonomous SRE Agent With Four Memory Layers

It's 3:14 AM and PagerDuty is buzzing. Prometheus has fired a critical alert: the payment service's error rate jumped to 34%, pods are dying with OOMKills and the SLA breach window opens in six minutes. The options are to haul yourself out of bed and SSH into a cluster, or to let an agent deal with it.

AegisOps is an autonomous SRE agent built for exactly that moment. It diagnoses, acts on and resolves production incidents without a human, and records every decision it makes and the reason for it. It isn't a chatbot wrapped around kubectl. It has real memory, layered autonomy controls and a way to learn from incidents it has already handled. The stated goal was never to replace the on-call engineer, but to give them a tireless first responder that already knows the runbooks, remembers the last ten incidents and checks policy before touching production.

Why ReAct on LangGraph

ReAct (Reasoning plus Acting) fits incident response naturally. A plain chain executes steps blindly, whereas a ReAct agent produces a thought, picks an action, observes what happened and loops. That's the right habit for an outage, where the useful questions are what's known, what's being observed and what to try next.

LangGraph provides stateful graph execution. Every node sees the complete AegisState, a TypedDict that holds messages, memory, incident details, policy decisions, tool calls and verification results. Because the state is checkpointed to PostgreSQL after each step, a run can be paused, examined and resumed.

The graph has four nodes, kept deliberately simple to avoid overengineering:

• • memory_node goes first and loads everything the agent will need before any reasoning begins.
• agent_node is the LLM reasoning loop.
• tools is LangGraph's stock ToolNode, which carries out whatever the model asked for.
• resolution_node wraps up the incident: it saves memory, works out MTTR and sends the Slack message.

Four layers of memory

Most agent systems skip memory architecture or settle for two layers. A human SRE brings four kinds of memory to an incident, and AegisOps reproduces all of them.

1. Episodic memory (pgvector, cosine similarity) holds past resolved incidents as embeddings. When a new one arrives, the three most similar earlier cases are fetched and injected into the system prompt with their root cause, the actions taken and the outcome.
2. Semantic memory (RAG over a knowledge base in pgvector) holds chunked runbooks, architecture docs and postmortems. The retrieve_runbook tool searches them semantically, giving the agent the team's institutional knowledge.
3. Procedural memory (YAML policies, a rule engine and Redis) covers hard rules about what the agent may do at each autonomy level. They're loaded from params.yml into Redis at the start of the graph and evaluated before any action that changes something. Consider it the agent's trained reflex for safe operations.
4. Working memory (the AegisOps state and the LangGraph checkpoint) covers the in-flight state: messages, current thoughts, tool calls, health-check results and retry counts. Checkpointing at every node means context survives restarts.

Episodic memory in practice

On entry, memory_node calls retrieve_similar_incidents with the symptom list of the current incident. A 768-dimensional sentence encoder embeds those symptoms, and pgvector's cosine distance compares them against the IncidentMemory table. Past incidents scoring above a 0.72 similarity threshold are formatted and placed in the system prompt, ahead of the agent's first thought.

After a resolution, resolution_node calls save_incident_memory, so the agent writes its own memories. The embedding comes from the symptoms joined with the root-cause hypothesis, so later incidents with similar symptoms will find this one.

A dedicated vector database such as Pinecone or Weaviate was an option. Keeping everything in PostgreSQL removed one infrastructure dependency and gave transactional consistency between incident records and their embeddings. At this scale pgvector is more than enough.

Autonomy levels L0 to L3

This was the hardest design question: how do you give an agent real power over production without risking a catastrophe? The answer is a layered autonomy model in which every mutating action is policy-gated before it runs.

Each action tool (restart_deployment, scale_deployment, update_deployment_image) begins with a call to retrieve_policy. The policy engine weighs the action, the service, the severity and the current autonomy level against the rules in params.yml. If the answer is no, the tool returns a denial right away and the agent has to find another route, usually escalation.

The requires_approval field deserves attention. Even at L3, some destructive actions, such as rolling back the image on a critical payment service, still need explicit human approval. The agent posts to Slack and waits. That gives the on-call team confidence that the riskiest operations keep a human in the loop.

The sixteen tools

AegisOps ships sixteen tools in four categories. Six stand out:

1. get_logs is the evidence collector. It takes the final 500 lines of a service's log file, keeps the requested levels and scrubs each line of IP addresses, UUIDs and plain numbers. Two refusals from different hosts therefore count as one pattern. A Python Counter ranks the patterns and returns the most frequent ones. Terminal readers and the agent's reasoning loop get the same structured result, and no Elasticsearch cluster is involved, just a log file and a handful of Python lines.
2. get_pod_status asks the Kubernetes CoreV1Api for the pods behind an app={deployment} label and reports restarts, readiness and container state. When the cluster is unreachable it degrades politely and sends the agent to get_logs instead.
3. restart_deployment does most of the heavy lifting. Patching the kubectl.kubernetes.io/restartedAt annotation starts a rolling restart, then it waits for every replica to become ready and runs a health check. An optional drain step keeps traffic away during the restart.
4. update_deployment_image is the emergency rollback. It rewrites the container image in the spec, waits for the rollout to finish and checks pod health afterward. Since it's the strongest lever in the toolbox, critical production incidents need L3 autonomy to use it.
5. check_service_health merges pod readiness from Kubernetes with a live error rate from Prometheus. It answers True only when every pod is ready and 5xx responses stay under 5%.
6. retrieve_runbook searches runbooks, postmortems and architecture notes by meaning and returns the four best chunks at similarity 0.75 or above, which land in the system prompt under a "RUNBOOK CONTEXT" heading.

One incident, start to finish

Here's what the agent did for the payment-service OOMKill at 3:14 AM:

[MEMORY] Loading episodic context for payment-service...
[MEMORY] Found 2 similar past incidents (similarity: 0.89, 0.76)
[MEMORY] Loaded 3 procedural policies for payment-service/critical

[THOUGHT] Incident: payment-service OOMKill, 3 pods CrashLooping.
Past incident INC-2801 had identical symptoms — root cause was
a memory leak in the gRPC connection pool after a deploy.
Starting with get_logs to confirm before acting.

[ACTION] Calling tool: get_logs with {'service_name': 'payment-service', 'log_level': 'error'}
→ top_errors: [{"pattern": "out of memory: kill process", "count": 47},
{"pattern": "grpc connection pool exhausted", "count": 23}]

[THOUGHT] OOM confirmed. gRPC pool exhaustion matches INC-2801 pattern.
Retrieve runbook before acting — want the fix procedure.

[ACTION] Calling tool: retrieve_runbook with {'query': 'payment service grpc oom memory leak fix'}
→ [runbooks / payment-service-oom.md] Recommended action: rolling restart
resets connection pool. If recurs within 1h, rollback to previous image.

[THOUGHT] Runbook confirms rolling restart. Checking policy for autonomy L2...

[ACTION] Calling tool: restart_deployment with {'service': 'payment-service',
'deployment': 'payment-worker', 'namespace': 'production',
'severity': 'critical', 'autonomy_level': 'L2'}
→ policy: ALLOWED (L2 ≥ min_level L2 for restart_deployment)
→ draining pods... restarting... polling rollout...
→ health_check: healthy=True, pods_ready=3/3, error_rate=0.008

[THOUGHT] Restart succeeded. Error rate 0.8% — below 5% threshold.
Verifying with check_service_health before closing.

[ACTION] Calling tool: check_service_health with {'service': 'payment-service',
'deployment': 'payment-worker', 'namespace': 'production'}
→ healthy=True, pods_total=3, pods_ready=3, error_rate=0.007

[THOUGHT] Service fully recovered. Writing audit log and resolving.

[ACTION] Calling tool: write_audit_log ...
→ audit_log written: restart_deployment for incident INC-2847

RESOLVED: payment-service recovered after rolling restart. Root cause: gRPC
connection pool OOM — matched pattern from INC-2801. All 3 pods healthy,
error rate 0.7%. MTTR: 187 seconds.

[MEMORY] Saving incident INC-2847 to episodic memory...
[NOTIFY] Sending resolution to Slack channel: #all-aegisops ✅

The whole thing took 187 seconds, from alert to resolution in just over three minutes, with nobody woken. That's the figure now reached routinely on incidents resembling earlier ones. The agent began by looking at what worked last time, as a senior engineer would, which is episodic memory doing its job.

That trace was reconstructed for clarity. A real run on a modest local setup, a single Kind cluster and no Prometheus, looks different. The incident was triggered by a simple PowerShell webhook call:

Invoke-RestMethod -Uri "http://localhost:8000/webhook/incident" `
  -Method Post `
  -ContentType "application/json" `
  -Body '{
    "source": "webhook",
    "severity": "critical",
    "service": "payment-service",
    "symptoms": ["High 500 errors", "DB connection timeouts"],
    "autonomy_level": "L2"
  }'

The agent ran it end to end. It retrieved similar incidents and formed a hypothesis that a traffic spike was overwhelming pod capacity, matching a past case in memory. It called get_pod_status and saw 2 of 5 pods ready, scaled the deployment to 4 replicas, confirmed with check_service_health and closed the incident as RESOLVED. A Slack message landed in #all-aegisops and the incident was written back to episodic memory.

That's tamer than a gRPC out-of-memory crash at 3 AM, yet the loop is unchanged: the same Thought, Action, Observation cycle, the same policy check before scaling and the same memory write at the end. Only the surroundings are thinner. The agent can't tell a laptop cluster from a 50-node fleet. Change the kubeconfig, aim Prometheus at real metrics and the same code works on the real thing.

Slack as the output channel

Notable actions send a Slack message. When an incident closes, the resolution node writes to #all-aegisops with a severity color, the MTTR and the whole resolution summary, which means the on-call engineer wakes up to a green checkmark and not a pager alert.

The stack

The whole thing runs on a modest stack. PostgreSQL with pgvector handles episodic memory and the LangGraph checkpoints. Redis holds policies and is reserved for future rate limiting. Prometheus feeds real error rates into check_service_health, and Grafana provides dashboards.

version: "3.9"

services:
  postgres:
    image: ankane/pgvector:latest
    ports:
      - "5445:5432"
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: admin123
      POSTGRES_DB: aegisops
    volumes:
      - postgres_data:/var/lib/postgresql/data

  redis:
    image: redis:alpine
    ports:
      - "6380:6379"

  app:
    build: .
    ports:
      - "8000:8000"
    env_file: .env
    depends_on:
      - postgres
      - redis

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./infra/prometheus.yml:/etc/prometheus/prometheus.yml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"

volumes:
  postgres_data:

The app is FastAPI under uvicorn. Incidents arrive at a webhook endpoint that builds an IncidentEvent and calls run_agent(). Execution is async, so several incidents can be processed in parallel without blocking. The Kubernetes client is kubernetes_asyncio, which keeps cluster calls non-blocking.

One gotcha: get_pod_status and check_service_health load the Kubernetes config when the module is imported, not inside the async function. If the cluster is unreachable at startup, they fail silently and return warning messages, so make sure KUBECONFIG or the in-cluster service account is configured before the app starts.

The audit trail

Each tool call gets recorded in a ToolInvocation TypedDict: what ran, with which parameters, what came back, whether it succeeded, how long it took, when, and which policy decision allowed it. The records pile up in state["tool_invocations"] during the run and are saved to episodic memory at resolution. A separate write_audit_log tool emits a structured JSON line for every notable action.

The result is that any incident can be replayed afterward: the sequence of steps, the permissions in force and the reason behind each one. Compliance teams ask for precisely that kind of record.

One design choice carries the rest. Each mutating action reports back the policy decision together with the health-check result, so the agent learns three things at once: whether the call worked, whether it was permitted and whether the service is healthy now. All three shape its next thought.

What's still missing

Everything above runs end to end. Pushing further was limited only by a local machine straining to run Ollama next to the full system. With more resources, three gaps come first.

The approval flow is the most pressing gap. approval_status already exists in the state and requires_approval is already checked, yet nothing actually pauses. The plan is to send a Slack button, halt the graph and resume only after a person clicks, before any L3 action goes ahead. LangGraph can suspend a graph mid-run through its checkpoints, so what remains is plumbing, not redesign.

Before-and-after metric comparison is the second. check_service_health is a point-in-time snapshot after the fix. What the agent needs is a Prometheus delta, the error rate five minutes before against five minutes after, to be confident it fixed something and didn't just benefit from timing.

Runbook self-authoring is the long-term goal. At the moment the two stores of knowledge, past incidents and runbooks, never feed each other. After an unusual fix, the agent ought to write a fresh runbook chunk and add it to the knowledge base, so retrieve_runbook returns it the next time. That would tie the two memory layers together.

What the project taught

The hard problem with autonomous infrastructure agents turns out not to be the model's reasoning but the safety structure around it. Memory layers, a policy engine, an audit trail and approval gates are what make a team willing to leave a tool alone at 3 AM.

AegisOps takes a stance: a structured state machine with clear permission boundaries beats a free-form agent that improvises. That stance has held up so far, with a richer incident log and an agent that improves after each new case. Checkpointing in LangGraph, the clean split between diagnostic and action tools and the TypedDict state make the system easy to inspect, debug and audit when something goes wrong and the question is why the agent chose what it did.

If you're building something similar, invest in memory architecture early. An agent with good memory is worth ten times one with a slightly better prompt. Transformers needed attention, and agents in the real world need context.