A Production Harness for Claude Code: Six Layers From Slack to Merged MR

Put a coding model on its own and you get a brain in a jar. It writes code and calls functions just fine, yet it won't see your Slack DM at 3am, retry a CI job that went red, sort out the merge conflict that just appeared on its open MR, or recall that a reviewer asked something yesterday and nobody replied.

What turns the model into something useful is the harness: the runtime scaffolding that supplies senses, hands and memory. Its parts are event intake, agent orchestration, durable state, loops that heal themselves, observability, and a way for the human operator to steer. The system described below is a production one and no research demo. It watches a Slack channel, opens merge requests against five internal repositories, answers reviewer comments overnight and repairs red CI runs without being asked.

It is an internal, multi-agent pipeline built on Claude Code. It has been running nonstop for several months, taking tasks from Slack and dispatching real engineering work. What follows is the top-down picture: what it does, how the parts connect, and the production scars that shaped the design.

The sections run in this order: why a harness at all, what this one does, the three gaps it closes, six layers (event ingestion, agent orchestration with a closer look at the autonomous loop and the self-improvement layer, persistent state, self-healing, observability and human control), a few extras beyond the layers, an end-to-end walkthrough and a comparison with other harnesses.

Why a chat window and some tools aren't enough

When plugins for chat assistants showed up in 2023, plenty of teams tried the obvious shortcut. Put a chat UI in front of an LLM, hand it a few function-call tools and label the result "the engineering agent." That setup nearly works in a demo and never survives production. Within a week three failure modes show up.

1. Context vanishes when the operator closes the tab. Memory has to outlast a single chat session. A genuine engineering task spans days: open a ticket, branch, draft, get reviewed, answer comments, wait for green CI, merge. No single model call holds all of that.
2. Nothing reacts to outside changes. A reviewer comments at 4pm. CI breaks on the third commit. A teammate replies in the thread. The agent has to wake up on those events, rather than polling forever or waiting for a person to prompt it again.
3. Failures are never recovered from. The agent pushes, CI breaks, and the operator has to return the next day and explain the failure from scratch. Or the agent's session with the identity provider expires halfway through a task and the whole pipeline dies without a sound. Both happen daily, and engineers end up paged on evenings and weekends for work the agent was meant to own.

Fixing these is the harness's job. It is also what sets a plain chat tab apart from Claude Code itself, from Cursor's background agents, from Devin, or from the system described here. The model underneath is close to interchangeable across all of them. The harness wrapped around it is what differs.

The model reaches the world only through the harness, and the harness does nearly all the engineering that makes an LLM behave like a system.

The job the harness was built for

The starting problem was concrete. A Slack channel, call it #support, collects about 30 tickets a week. Each needs an investigation, and usually a code change in one of five repos afterwards. A ticket stretches over days. Reviewers comment on MRs, CI flakes, and some investigations turn out to repeat last month's. The team carrying this load kept getting paged in the evenings and at weekends, and nothing improved because the same ticket shapes came back every cycle.

The harness has three responsibilities, ordered by how deeply it intervenes:

1. Investigate. When a new ticket lands in #support, follow the thread, collect context across repos and post a structured analysis in that same thread.
2. Fix. Once the operator approves, open a merge request with the proposed change, answer reviewer comments, watch CI, repair it when it breaks and carry the MR to merge.
3. Self-improve. Every closed case feeds back. If a pattern repeats, say five tickets in one area all touching the same config, the harness writes up the gaps and proposes changes to itself.

That third duty is what separates this from a fancy ticketing bot. The harness's raw material is its own source code, and the LLM can write to it through pull requests that pass human review.

Six logical layers make up the system, and each gets its own section below.

The three gaps a bare model can't close

One idea underlies the whole build: a stateless, purely reactive LLM isn't a production system. The six layers exist to close three gaps that the model can't close alone.

The first gap is reactivity. A model can't wake itself up. Layer 1 (event ingestion) handles that, since Slack mentions, GitLab CI results and PagerDuty pages all flow into one dispatch queue. The LLM never polls. It gets called.

Persistence is the second gap. Models forget everything between sessions and lose context when work moves between machines, and they can't tell the difference between having addressed something and a commit that actually reached origin. Layer 3 (persistent state) covers this: in-memory structures dedupe within one process, local JSON files hold the operational maps, and git-synced workspaces carry durable case state. Session continuity across machines, described later, grows from the same idea, which is that state belongs in git and never inside a running process.

The third gap is quality. Left unstructured, an LLM reasoning loop writes confident conclusions from thin evidence, calls investigations finished before checking every reachable source, and produces MR descriptions that read as machine-written from the first sentence. Layer 2 (agent orchestration) and the investigation loop address it with structured output contracts, a quality gate on every iteration, an adversarial review once the loop ends, and self-healing loops around the outer cycle of code change, CI and merge.

Compounding is what ties the layers together, because every closed case speeds up the next. Approved investigations become knowledge-base entries. Approved MRs turn into patterns the gap analyzer can point to. A finalized case opens an improvement MR aimed at the harness itself. With each cycle, fewer kinds of tickets get handled badly.

Layer 1: how events wake the system

Three outside signal sources can wake the harness:

• Slack messages, meaning channel mentions, thread replies and admin commands sent as a DM to oneself.
• GitLab activity, meaning MR review comments, pipeline results and pipeline failures.
• PagerDuty alerts, used when an on-call page mentions a ticket the harness has already seen, so the earlier context can be surfaced.

The three differ in latency and reliability, so each gets its own ingress path.

Two practical details never appear in the textbook version of this diagram.

Eventual consistency. Slack's conversations.history endpoint can lag by anywhere from 1 to 30 seconds. A poller that sees an empty result and naively moves its cursor forward will skip, for good, any message whose ts falls inside that window. So the cursor is frozen on empty polls and the next attempt starts from the same oldest timestamp, repeating until at least one message comes back. A per-message-ts dedup set filters the duplicates those retries produce.

Socket Mode plus a poller, belt and suspenders. Socket Mode delivers thread replies with sub-second latency, but WebSocket connections drop. The poller acts as the safety net, catching whatever Socket Mode missed, and both share a socket-dedup file so no message is dispatched twice. Take that away and anything that arrives during a dropped connection disappears silently, with no signal to the operator that it was missed.

The mr-monitor cron loop behaves differently. GitLab doesn't push review comments into the pipeline, so polling is the only option, and scaling the polling itself proved the hard part. Each additional long-lived MR the harness takes on adds to a 10-minute loop that piles up cursors for dead threads, and each one costs a Slack API call per cycle. Left alone, they caused sustained 429 rate-limit storms that starved the polling that mattered. The fix evicts a cursor automatically when the upstream API answers thread_not_found, so it drops out of the next cycle's loop. Before the fix, a cycle made 12 retries for hours at a time. After it, none.

Layer 2: how agents are orchestrated

The LLM actually runs here. The harness launches workers as transient systemd units, and each is handed a JSON payload naming the workflow to run, the case_dir to use and the thread it belongs to.

Each pipeline amounts to one long Claude Code run (the brain) that gets a curated prompt and a writable case_dir. That directory serves as the agent's scratch space, holding the case's TASK.md, a growing pile of followup_transcript_*.md transcripts, any gap_report*.md reports the agents produce, and intermediate JSON artifacts.

How the agents are sequenced is part of the design. Instead of one agent doing it all, several specialized ones are chained together, and each hands off at a fixed point:

• oncall_run performs the first investigation, up to 20 reasoning iterations with a quality gate, ending in an explicit self-critique. Its output is an ANALYSIS.md describing what's wrong and a TASK.md describing what to do.
• case_followup is the long-lived chat agent. An operator can reply by DM on a case thread for days, and every reply re-invokes case_followup with the entire conversation so far. It also calls gap_analyzer as a sub-agent every N replies, and that loop produces the gap_report*.md files that point at the harness's own code (the self-improvement loop is covered further down).
• finalize_case is the consolidator. Once a case has been idle for four hours, or an operator runs admin: finalize <case>, it gathers every gap report, drops the false positives and opens a single MR with the fixes that survived. That MR targets the harness codebase and not the ticket's original repo, which is the self-improvement loop in concrete form.
• dev-agent is the one that does the hands-on work. Inside its own per-case git worktree it edits the code, pushes, watches CI, repairs CI when it breaks and answers reviewers. It runs in three modes: address_review, fix_ci and task (the original implementation).

This graph has a pair of details that are easy to overlook.

First, nothing touches the main checkout, because all work happens in worktrees. That lets several cases run side by side without colliding. The poller and the dev-agent dispatcher put the worktree path in the payload, so the LLM never gets to see the main checkout at all.

Second, case_dir anchors everything. State files in ~/.harness/ (see the persistence section) tie each Slack thread to a case_dir, and the agent resolves every path relative to it. Everything else gets simpler as a result: move the directory, update the map, and each workflow follows along.

The graph shows what runs. The next section explains how it runs, meaning the mechanics that carry a Slack command through to a finished investigation.

The autonomous loop, step by step

If someone asks what this system actually does, the shortest honest answer is: an operator types one Slack command, and from then on a closed loop of repeated reasoning, self-critique and tool use runs unattended until the work is finished or the system admits it can't go on. Nearly every interesting design decision hides inside that sentence, and the rest of this section opens it up.

Inside the harness sit four distinct agent loops, and each is shaped to fit its own sort of problem:

That picture gives the shapes. The substance comes next, with each mechanism in the order it runs. The diagram maps onto all of them, and its nodes make sense once the sections below have been read. The gate codes it uses are G1 to G4, N1 to N3, N5, A1 to A3 and D1 to D15.

Before the loop: the pre-commitment lock

Two safeguards run ahead of the first Claude call, and the first is structural. Before any investigation iteration happens, the on-call pipeline has Claude commit to a hypothesis_slate. That slate lists at least six candidate explanations plus four or five criteria for picking among them, and it's locked into kickoff_precommitment.json, which nothing may change afterwards.

Of everything in the system, this lock is the safeguard with the greatest methodological weight and the least visibility. An unconstrained LLM reasoning loop tends to settle on whatever its first evidence favors, then reads all later evidence as confirmation. The lock can't stop that pull. What it does is force every investigation to begin from the whole hypothesis space and to record how it got from there to its conclusion. The adversarial reviewer (Phase B) then asks a harder question than internal consistency: is this the conclusion the original slate would have predicted, given the evidence?

The second safeguard is the empirical anchor (Phase C). Only on iteration one, a few OS-level commands run with no LLM in the loop: journalctl error tails, disk and memory snapshots and a search for log files. The results go into the opening investigation prompt as raw evidence, giving Claude a factual picture of the machine before any hypothesis exists. The step never blocks, silently swallows errors and takes under a second. It earns its place because the first reasoning step then begins from ground truth rather than from the problem statement alone.

Structured output: the completion report

Each iteration ends with a parseable JSON contract, the completion_report, and the harness reads it to decide what happens next. Its shape is fixed:

{
  "status": "IN_PROGRESS" | "BLOCKED" | "COMPLETE",
  "confidence": 78,
  "open_questions": ["..."],
  "unchecked_sources": [{"name": "...", "access_status": "..."}],
  "contradiction_register": ["..."],
  "assumption_register": ["..."],
  "adjacent_problems": [{"summary": "...", "status": "...", "blocker": "..."}],
  "draft_response": "the user-facing Slack message"
}

Free-form prose may surround the block, but the gates evaluate the parsed report and the state files persist it. Trading some prose freedom for a parseable contract was the biggest single lever found for keeping LLM output usable inside a pipeline. Skip the schema and every later step has to dig facts back out of prose. Keep it and the gates become ordinary deterministic Python.

Quality gates that give confidence a meaning

A confidence score would be marketing noise if Claude could type confidence: 95 and move on. So every iteration's output passes through quality gates that refuse to let the investigation advance unless the evidence and the stated confidence agree.

Three families of gates run after each Claude turn:

• G-gates check logical consistency, so that reasoning doesn't contradict itself between rounds. The clearest case is G1, which fires when open_questions grew during the round yet confidence went up, because a growing denominator cancels any upward move. The family also checks that tension language in the prose registers as a contradiction, and that neighboring problems are tracked as valid status objects instead of placeholder strings. There are four.
• N-gates check structural completeness, making sure artifacts are fully formed and not stubs. N1 requires every executable fix action to be paired with a verify_action that gives a description, an expected outcome and timing. A fix proposed without a way to verify it counts as unfinished work. There are three.
• A-gates impose mechanical ceilings that prose can't talk around. A1 computes a hard confidence cap as 1.0, minus 0.08 per open question, minus 0.05 per unchecked source, and whatever Claude writes, the gate clamps the number arithmetically. Other A-gates block COMPLETE while adjacent problems stay open, demand a non-empty assumption register and insist that at least one proposed action maps straight to the root cause. There are seven.

Two spot-checks sit on top. EQ1 makes root-cause nodes cite hard evidence (anything other than INFERRED) or accept a 60% cap, and P7 lowers the ceiling by 7% for every adversarial dimension marked FAIL, down to a floor of 40%.

The framework totals 19 functions. A handy way to remember them: G-gates react to what Claude writes, N-gates to what it omits, and A-gates to what it asserts.

A violation becomes a pending_guard_notes entry, which gets placed at the head of the next iteration's prompt under the heading ## SYSTEM QUALITY GUARD NOTES (from previous iteration), with an explicit instruction to address every note first. That's the forcing mechanism. The gates don't block the model. They make its earlier violations the first thing it reads next round. Politeness on the model's part doesn't matter, and the structural enforcement is what moves the investigation along.

A context-pinning step (A7) also runs at the end of every prompt builder, whether or not anything was violated. It reattaches the unresolved contradictions and unverified assumptions from the current report, word for word, to the bottom of the next prompt. That counters the well-known attention fade, where a model notes a complication in round 3 and has quietly dropped it by round 7.

Confidence as a concrete number

The most consequential design choice is that the agent assigns itself a confidence: <int> on a 0 to 100 scale in every iteration's structured output. The number isn't decoration, because it controls real behavior:

• Below 70, the loop can't be left. The pipeline forces up to three more rounds and injects a guard note saying the agent claimed to be done while sitting under the bar.
• At 70 or above with status: COMPLETE and no open_questions, the gate opens to the adversarial review phase (red team against blue team).
• At 95 or above, the agent may auto-run non-destructive actions such as a Jira comment, a status transition or a ticket assignment. Under 95 those stay as drafts for the operator to decide on.
• A drop of more than 10% across two consecutive rounds ends the loop with the reason degrading. The agent is going backwards, so it stops and tells the operator.

Each case's full confidence trajectory is written into ANALYSIS.md as a sparkline such as 45% → 62% → 78% → 80%. Operators glance at it to triage whether the result deserves trust, and the harness's own self-improvement loop reads it when choosing which past cases to re-run with a different prompt.

Three ways the loop refuses to stop early

In the diagram, three branches reach ready_for_adversarial or blocked without the loop ending. Each is a force-continue rule: the system overrules the exit signal and injects a guard note in its place.

• CONFIDENCE_FORCE applies when status=COMPLETE and open_questions=[] but confidence is under 70. The model declared victory while signaling doubt through the number. Up to 3 further iterations are forced, and they count inside the 20-iteration budget instead of adding to it.
• CAUSAL_CHAIN_FORCE applies when status=COMPLETE and confidence is at least 70, but causal_chain_complete isn't true. That's a symptom-level finding with no traced root cause. It forces up to 3 more iterations, again inside the 20-iteration budget.
• G_BLOCKER_FORCE applies when status=BLOCKED while reachable technical sources were skipped or declared unavailable by inference and not because of a real error. An investigation can't exit as blocked without truly exhausting what it could get to. It allows up to 3 continuations, counted inside the same budget.

Between them these stop the three most common premature exits: confident-sounding uncertainty, symptom-level completion and imagined unavailability.

The adversarial reviewer

When the on-call pipeline declares itself done at confidence 70 or higher, control passes to a separate Claude invocation playing an adversarial reviewer. This reviewer doesn't see the investigator's prompt. It receives the analysis text and a checklist of 15 evaluation dimensions, D1 through D15, covering assumption validity, the evidence chain, causal reasoning, alternative hypotheses and more. It returns a structured verdict:

There are three ways out of the review:

1. Critical issues go back to the investigator. Their notes are injected into the following iteration as pending_guard_notes and the loop restarts, with the investigator now holding explicit word that the reviewer believes something was missed.
2. If there are only minor issues on a first pass, the loop goes round one more time with the minor critique attached.
3. Two consecutive minor-only passes, or one clean pass, end the loop with complete, and the investigation is published.

Making the adversarial step a separate invocation with a different prompt is deliberate. The same model in a different role catches a meaningful share of the self-flattery that single-prompt iteration lets through. On one case a confidence claim dropped from 90 to 65 between the investigation and its review, with the reviewer pointing at the very assumption the investigator had glossed over.

A second review runs alongside it, built differently: the red team review (G5). The adversarial reviewer reads the full investigation narrative, while the red team reviewer is shown only the original problem statement and the final conclusion. It gets no intermediate reasoning, no evidence trail and no earlier rounds. From those two items it forms an independent hypothesis and flags wherever the investigation's findings part ways with what an outsider would have guessed.

The aim is cognitive isolation, not extra coverage. The adversarial review asks whether the investigation is structurally complete (D1 to D15). The red team asks whether the conclusion could have been reached without the investigation's framing: is the reasoning truly grounded in the problem, or did the investigation anchor on an early finding and let the conclusion follow the anchor instead of the evidence? Both run as separate Claude calls, and both must pass before a case is marked COMPLETE.

Five honest ways to end

The investigation loop can finish for five reasons. Whichever one fires is logged and shown to the operator, as in lines like these:

exit=complete       confidence=89%  iterations=12  elapsed=13min
exit=blocked        confidence=62%  iterations=4   elapsed=4min    (cannot proceed without info)
exit=stalled        confidence=58%  iterations=7   elapsed=22min   (two rounds, zero new facts)
exit=degrading      confidence=42%  iterations=5   elapsed=8min    (dropped >15% twice)
exit=timeout        confidence=73%  iterations=18  elapsed=50min

Of the five, blocked is the one an operator can act on most directly. The agent has recorded explicit blockers in unchecked_sources or adjacent_problems, and a human has to make a call to clear them. The other four are technical failure modes that the harness surfaces and does not retry blindly.

What Claude can reach from inside

Tool boundaries count here, because the confidence gates partly depend on what Claude can actually touch. A gate telling the agent to check its unchecked sources means nothing unless the tools to check them exist. Each pipeline launches a Claude Code subprocess with --permission-mode bypassPermissions. No inline tool allowlist appears in the prompt. The boundary is whatever is wired into the Model Context Protocol (MCP) catalog, which spans five external systems:

A handful of constraints deserve a mention, because they influence behavior more than anything on the model card:

• The investigation pipeline explicitly bans Bash (--disallowedTools Bash), which heads off git race conditions, because two Claude processes editing one working tree at once would corrupt state. The dev-agent pipeline does get Bash, but it runs in an isolated worktree.
• Slack write tools are blocked in every pipeline invocation. send_message_to_self and edit_message_to_self are disallowed, so Claude can't post to Slack itself. All outbound communication passes through the draft_response field of the structured completion report, which the harness posts only after it has validated the exit conditions.
• bypassPermissions doesn't mean unrestricted in practice. Each pipeline's framing prompt encodes hard rules. ANALYSIS.md can be edited but message.json can't. Writes outside case_dir are reverted automatically through git diff after the run. Production database writes wait for explicit operator confirmation, per command. Safety lives in the framing layer plus a post-run boundary check, and not at the tool-call layer.
• Cascading fallbacks live in prompts and aren't hardcoded. If search_datadog_services comes back permission-denied, the prompt directs the agent to switch to mcp__glean_default__search, using Jira keywords as a stand-in. The fallback chain is part of how the SOP gets enforced and has no place in the runtime.
• Read-only tools run in parallel and write tools run in sequence. The streaming-mode subprocess parses tool_use events as they arrive and dispatches up to 10 concurrent read operations (Datadog queries, Glean searches, file reads). Anything that writes, whether Bash with side effects, edits, MR comments or database writes, goes one at a time. A single investigation iteration can therefore fire six parallel log queries and still keep writes ordered.

One skeleton, four sizes

Every pipeline in the harness runs some variant of the same skeleton:

The on-call investigation pipeline runs up to 20 iterations of it. The task-analyze phase gets up to 8. The implement phase runs the skeleton once but loops its build step three times. The dev-agent has no inner loop and runs once per spawn, but mr-monitor effectively moves the iteration outside: it spawns a fresh dev-agent each time CI fails, up to three consecutive times, before handing off to a human.

The uniformity is intentional. The four loops exist because the problems differ, yet they speak the same language: confidence is a number, an exit has a name, self-critique is a separate Claude call, and escalation means a Slack DM in the thread that started the case. An operator learns the shape once and then recognizes it everywhere.

The self-improvement layer

This closes the loop sketched earlier, in which the harness modifies its own substrate under human review. Two loops do the work, on different timescales.

Loop 1 repairs structure case by case, from gap to PR. After an investigation closes, gap_analyzer looks for places where the harness underperformed: skipped data sources, wrong tool choices, pipeline bugs, missing tests and badly formatted MR descriptions. Category and confidence filters thin out the gaps before finalize_case receives them. Once a case has sat idle for four hours, finalize_case starts patch_suggester in a dedicated worktree and produces one PR against the harness's own codebase, touching code files, prompt templates or SOP sections. Merging it takes a human. Writing it doesn't.

Loop 2 reinforces behavior across cases, from failure mode to SOP suggestion. At the end of each case, the auto_retrospective step records which failure modes occurred, sorted into three types: data-gathering failures (were reachable sources skipped?), quality-gate violations (did a confidence claim contradict the evidence?) and analysis-completeness failures (did root-cause tracing stop short?). Records accumulate per project in the case registry. Every week a cron job named knowledge-updater tallies the last ten cases and, when a mode crosses a threshold, drafts an SOP suggestion: 3 of 10 yields a suggestion and 6 of 10 yields an alert. The suggestions are concrete. For example, "skipped direct DB query" fired 3 times this month, so the SOP now requires the direct table query before any schema inference. Humans review and apply them, and nothing is auto-applied here by design, since automated decisions that compound their own errors are a known hazard in self-modifying systems.

Personalization follows from this. The SOP encodes the harness's picture of how a team investigates, and it starts out generic. Over time the kb-updater builds a profile of one project's failure patterns from the cases and repeats. By 20 to 30 cases, that project's SOP no longer resembles the generic template: it records which data sources keep getting skipped, which tools get misused and which investigation framings regularly miss in that specific codebase. Projects keep separate SOPs, so Team A's mistakes shape only Team A's rules and never contaminate Team B.

Durability is the other benefit. A context window lasts a single session, whereas a merged code change is versioned, reviewed, tested and permanent. An SOP update, once an operator has approved it, shows up in every investigation afterwards. The substrate keeps improving, so each new case runs on a better harness than its predecessor.

Layer 3: state that survives restarts

Orchestration is what runs per event. Persistent state is what remains when a process restarts, a machine reboots or the operator moves from a laptop to a cloud VM. The harness keeps its state in three layers with sharply different properties:

In-memory caches are small and rebuildable. They live in the poller and keepalive process and get reconstructed from local state after a restart.

Local state in ~/.harness/*.json is the operational map. The key files are:

• task-thread-map.json maps a Slack thread_ts to a JIRA id and an absolute task_dir path.
• oncall_thread_map.json has the same shape for on-call investigation cases.
• monitored-mrs.json lists the MRs the harness opened along with their CI status.
• patch-pr-state.json records per-case finalize status (merged, no_gaps, failed and so on).
• oncall-state.json holds the kill-switch flag, covered under Layer 6.
• socket-dedup holds the timestamps of recent messages Socket Mode already processed.

This layer belongs to one machine and isn't shared. It survives a restart, but if the machine dies it's gone.

Git-synced state is the durable layer. Case workspaces under case-workspaces/<ticket-id>/ are committed and pushed to a private git repo at every meaningful state change. A second machine that clones the repo can pick up the same conversation context, which is what provides cross-machine session continuity. An operator can chat with the harness from a laptop or a remote dev VM and lose nothing.

Memory consolidation later became a third job for git-synced state. A long conversation produces dozens of followup_transcript_*.md files. When enough activity builds up, a sub-agent reads all of them, rewrites TASK.md to carry the condensed state, and the workflow quits feeding raw transcripts into later turns. Skipping consolidation lets prompts outgrow the model's context window. Doing it risks compressing too hard and losing information, and that really happened: a 14KB document shrank to 587 bytes. Three protections came out of that incident: a prompt template that says "UPDATE not REWRITE," an output-size guard that refuses writes below 50% of the input size, and a per-case lockfile so two consolidation runs can't race.

The lookup path from a single Slack thread to the case's TASK.md is small but worth drawing:

Both state files hold the same kind of value, an absolute path to the case workspace. Any workflow resolves to a case_dir sooner or later and then works relative to it. When the workspace root was renamed midway through the project, one migration script handled it by rewriting the absolute paths in both maps while moving the directory.

Durable state is the foundation. The next question is what happens when operations built on it fail, and the answer is that the harness recovers from them and doesn't pass them on to the operator.

Layer 4: loops that repair themselves

This is the layer where the harness stops being a notifier and becomes autonomous. It has three loops, each closed with no operator involvement.

The first two are fully autonomous. The third is deliberately not. Refreshing the identity-provider session needs a hardware-bound MFA prompt, so the agent halts and asks for human action, putting a 30-second runbook in the DM. That's human-in-the-loop by design and no capability gap.

These loops share a subtle trait: each has a retry budget and can recognize what it already did. CI auto-fix is capped per MR, so three consecutive failures escalate to the operator. Reviewer auto-reply dedupes by author, which means that when the bot's latest note in a discussion is newer than the reviewer's, the loop stays quiet because the reply already exists. The identity-provider helper allows a 15-minute approval window. Remove those limits and the loops sometimes spiral. Early on, one MR with a flaky integration test picked up 80 no-op "fix CI" commits in a single afternoon before anyone noticed.

Recovering from single operations

The CI auto-fix and reviewer-reply loops described above recover whole workflows from start to finish. Beneath them, every primitive operation that can fail (a push, a rebase, an MR comment, a branch reset) rests on a separate, smaller three-layer design for per-operation self-recovery. It differs from the three-layer safety model in Layer 6, which is about restraint, meaning what the agent may do. This one is about recovery, meaning what happens when a permitted operation fails. The two are easy to mix up, so keep them apart.

L1 is a deterministic guard. Rule-based and always on, it responds in well under a second. For pushes it's _pre_push_rebase(repo_path, branch): right before pushing, the harness runs git fetch origin and git rebase origin/<branch>. A clean rebase lets the push go ahead. A conflict or failed fetch makes L1 abort safely, list the conflicting files and pass the case to L2. L1 can't be creative, on purpose. The rule stays small enough to reason about, and cheap enough, with no tokens spent, to apply to every single operation.

L2 is an agentic self-heal loop. When L1 fails to fix the situation, the harness starts a constrained Claude session (_self_heal_push_loop) and gives it everything about the failure: the git state, the conflicting files, the task that motivated the push and some tight red lines. Never force-push a protected branch, never discard unpushed commits and never choose a conflict side blindly. Three constraints make that safe:

1. Bounded retry. Each incident gets 3 attempts at most, and each Claude turn has a 4-minute timeout. Following every Claude action, the harness runs the real git push again. Claude's opinion on whether the conflict is gone carries no weight, and an exit code of 0 from git push is the only ground truth. If the next push returns non-zero, the loop logs a failed attempt however confidently Claude claimed success.
2. A confidence gate at 70 or above. This is borrowed unchanged from the oncall_run config of the on-call investigation pipeline. Claude rates itself 0 to 100 each turn, and below 70 it must bail instead of acting. A self-report under the floor doesn't get to touch the working tree.
3. The truthfulness invariant holds. A successful L2 run means an actual commit reached origin. The reply gate further down the chain, which sends the reviewer a "Done, updated" note, remains wired directly to git push exit 0 and never to anything Claude says about itself. L2 can't bypass the gate. It can only supply it with a real push.

L3 is operator escalation. If L2 bailed on low confidence or used up its attempts, the harness sends the operator a DM with the whole L2 trail, meaning each attempt, each action Claude took, each confidence rating it gave and the closing git status. The operator starts from a fully instrumented state and not from "something went wrong." The time scale is hours, not seconds.

Two principles stand out:

• L2 is per-incident and not always on. L1 runs on every push because it's cheap, while L2 burns tokens and fires only after L1 has failed, so the cost is gated by the rare case and not the common one.
• Ground truth comes from the wire. Whatever Claude believes about its own work, validation repeats the real operation: the git push exit code for pushes, an actual test run for test fixes and the reply gate's check of the push exit for MR replies. The agent's self-report is never treated as ground truth, because the harness re-checks against the real system.

That same wrapper suits far more than pushes. An operation qualifies when three things exist for it: a plain rule-based fix (L1), a tightly limited agentic retry (L2) and a clean way to hand off to a person (L3). This three-part shape separates an autonomous loop that recovers from failures it already knows from one that needs a human as soon as the world shifts beneath it.

Layer 5: four places to see what happened

The harness writes to four observability surfaces, and each one is tuned for a different reader:

• Live Slack updates. While an agent runs, a placeholder message in the case thread cycles through statuses (Reading TASK.md, then Searching codebase, then Drafting patch) and finally resolves to Done or Failed: <reason>. This streaming reporter edits one message in place, so threads don't fill up with noise.
• Structured logs at ~/.harness/logs/agent.log and ~/.harness/logs/error.log. Every spawn, chain dispatch and retry attempt becomes a structured event, and these files are what gets searched when something needs diagnosing.
• Slack DMs for terminal events, such as an MR opening, an MR merging, an expired identity-provider session or a CI escalation. They count as first-class operator notifications and are threaded under the case where one applies.
• A system log at ~/.harness/logs/system.log for cross-cutting events like deploy completions, watchdog actions and sync status.

A dashboard was left out on purpose. Slack and grep serve as the consoles. A dashboard becomes a third surface to keep current, and it's the first thing engineers stop reading once it goes stale. The streaming-message pattern gives enough operational signal that nobody has wanted a UI on top of it.

Watching the MCP connections

Five external MCP servers (Slack, Datadog, Jira and Confluence, Glean, GitLab) underpin the harness. Each carries its own OAuth token, refresh cadence and opaque ways of failing. When a token expires unnoticed, a Claude session starts replying "I don't have access to that tool," and the operator sees no error at all.

A small service called mcp-watchdog closes that hole. Every 10 minutes it polls each MCP's health endpoint and tries a no-op tool call. When that fails, it does the following:

1. It attempts a silent token refresh.
2. If the refresh fails, it posts a Slack DM to the operator, along the lines of "Atlassian MCP needs reauth, click here."
3. After two consecutive disconnects (the DISCONNECT_CONFIRM_THRESHOLD), it escalates with a :warning: reaction on the most recent ticket.

Absent this loop, an expired token gets mistaken for "the agent is dumb today," a kind of failure that's miserable to diagnose because agent.log never says "token expired." It says "tool returned empty result." Treating auth health as a first-class observability surface is something that should have been built sooner.

Layer 6: the operator's control surface

This is the most used surface and the one that was underbuilt at first. Every operator interaction goes through a single channel, the self-DM, and the commands fall into six families:

• Task: admin: task <jira> runs the entire analyze, implement and MR pipeline, and adding --auto makes it unattended.
• On-call: admin: oncall-run <url> kicks off an investigation from a Slack thread, while oncall-toggle turns the auto-monitor on or off.
• MR control: admin: mr rescan polls one particular MR again, and mr pause/resume switches the auto-reply loop on that MR off and on.
• Pause: admin: oncall pause [duration] mutes automatic dispatch for a while, and oncall resume/status ends the pause or reports where things stand.
• Finalize: admin: finalize <case> rolls up the gap reports and raises the improvement PR against the harness itself.
• Registry: admin: register-thread links a Slack thread to a case by hand, while close-case marks one as finished.

The pause kill switch took the longest to get right. Version one paused everything, including the admin commands, which left an operator who had paused the system unable to resume it. Version two paused too narrowly, so "stop the noise" wasn't honored. The current design depends on one precise distinction: pausing silences automatic dispatch triggered by other people's activity, and never the operator's own admin commands or in-thread actions. In the poller, the dispatch sites split into two groups. One group is automatic activity driven by others, and it's gated while paused. The other is the operator's own intent, and it stays live.

The calibration came the hard way. Shortly after the first pause shipped, a reviewer-comment auto-reply fired silently on a different case and went unnoticed, because the pause looked like it was in effect. The present design treats pause as directed silence: quiet the system while preserving the operator's agency.

A picture shows the split:

The rule is that the operator's own actions are never silenced, and other people's activity is gated while paused. That is everything admin: oncall pause controls.

Two more layers of safety

The pause kill switch is the most visible safety surface, though not the only one. Two others work without any runtime configuration, one in code and one through deployment convention:

The Slack channel allowlist deserves more weight than it seems to. A near miss early in the project involved a misconfigured Slack token that could write to internal team channels. The agent could have posted straight to teammates, skipping the operator, and nobody could have told those posts from a human's. Now each outbound chat.postMessage checks its destination against an allowlist before it's sent. The only permitted destination is the operator's self-DM, one hard-coded channel ID for the operator's own conversation with the bot. If the agent wants to reach a teammate, it hands the content back to the operator, who decides whether to forward it. That architectural rule keeps proving itself: the agent speaks outward only through the operator's mailbox.

Beyond the six layers: what makes it production

The six layers are the load-bearing structure. Several mechanisms sit outside that taxonomy and matter just as much. They separate a harness that demos well from one that runs without interruption.

Routing several projects from one YAML file

Two projects run on the harness today, each with its own operational rules, and adding a third takes zero code changes, only a new entry in agent-config/projects.yaml:

projects:
  team-a:
    sop: "team-a/SOP.md"
    kb: "team-a/KNOWLEDGE-BASE.md"
    dfr_threads_dir: "team-a/threads"
    task_dir: "task-workspaces"
    slack_channels: ["support-channel-a", "support-channel-b"]
    jira_prefixes: ["AAA", "BBB", "CCC"]
  team-b:
    sop: "team-b/SOP.md"
    kb: "team-b/knowledge-base/kb.md"
    dfr_threads_dir: "team-b/threads"
    task_dir: "task-workspaces"
    slack_channels: []
    jira_prefixes: ["XX", "YY"]

A resolve_project() function routes each incoming signal by precedence: Jira prefix first, then Slack channel, then the default. Every project has its own SOP file, KB file and directory tree, and prompt assembly for investigations picks up the right SOP and KB automatically according to which project the signal resolves to. That's what lets one harness instance serve two teams with two sets of rules.

The pattern generalizes. Any organization with several product teams on one harness can use the same approach: declarative config and no per-project code branches. It also enforces a clean discipline, since anything specific to one project has to live in that project's SOP file and never in code, because the code is shared.

Resuming a session on either machine

The operator works on two machines, a laptop and a long-lived cloud VM in the data center, and one Claude session must be resumable on both. That's hard because Claude Code stores session state on local disk by default, so after a restart elsewhere the session can't be seen.

The harness solves it with a git-backed session store:

The mechanism has three parts:

1. Sessions are tracked in git. Each session's .jsonl and .meta files are committed and pushed to a private repo at every meaningful state change, and that repo is the source of truth.
2. A background fetcher runs every 5 seconds on each machine to keep the local copy current. It pre-pulls context, so resuming a Claude session doesn't pay network latency on every keystroke.
3. Session UUIDs are matched by working directory. When Claude starts somewhere, the session-start.sh hook looks through the local .meta files for the entry recorded against that CWD and picks up its UUID. One UUID therefore follows the operator across machines.

The sync hook, user-prompt-submit.sh, fires after each Claude prompt and performs three operations in strict order, serialized with flock:

1. Fetch and reset. Running git fetch origin main && git reset --hard leaves the local active.jsonl holding the other machine's transcript.
2. Extract the delta before overwriting. It reads the watermark, which is how many lines have already been injected from the other machine, pulls out only the newer lines and saves them to a temp file. This must come before step 3. An earlier bug had the two swapped, so the injection kept reading the machine's own transcript back to itself.
3. Push your own transcript. It copies the local Claude session transcript to active.jsonl, commits and pushes, then sets the watermark to the local line count.

After the flock lets go, the delta saved in step 2 reaches Claude as a system message that says the other machine added N messages since the last sync. The next prompt then sees both machines' context.

A separate .linecount file holds the true total line count, because wc -l misleads once a transcript has been truncated for pushing. Transcripts get cut to 1,000 lines before a push to stay under the 25 MiB git limit. With no .linecount, every truncation would reset the watermark and the injection would redeliver thousands of lines already seen.

The flock-serialized git operation prevents index.lock corruption from concurrent pushes, which happened once when a stuck session start overlapped with a meta-enricher run. A throttle, configurable and defaulting to 30 seconds, skips the full sync on rapid keystrokes so there's no per-character git latency.

The broader idea worth borrowing is git as a state store. Git serves here as more than a code repository. It works as a CRDT-like durable log of agent state, with the operator's two machines acting as eventually consistent readers.

Memory consolidation is closely tied to continuity across machines. The memory_consolidate step, which rewrites transcripts into a clean TASK.md and ANALYSIS.md, doesn't run on a timer. A separate service, consolidate_trigger, checks each turn for four signals: a commit SHA or a decision verb in the transcript, a review thread that just closed, a case workspace with over 5KB of accumulated transcript, or at least an hour since the last consolidation. Any single signal triggers it. The reasoning is that information stops piling up after a decision, not after a set amount of time, and together the four signals estimate whether the conversation has reached a natural pause, so the operator never has to say so.

Worktrees are more than an implementation detail

Earlier it was said that everything happens in worktrees. That deserves expansion, because the choice behind it carries real weight.

The original design had the agent emit a diff and the harness run git apply in the main repo checkout. That produced two failure modes that kept recurring:

1. Brittle diff headers. Claude would emit a diff whose line numbers were right at generation time but wrong after a concurrent commit landed, and git apply would reject the entire patch.
2. A stuck main tree. If git apply --check failed midway, the main repo was left half-applied, and the next case to run on the same repo inherited the dirty tree.

worktree_manager is the fix. Ahead of any agent that must change code, it builds a sandboxed worktree under ~/.worktrees/case-<case_id>-<stamp>/, checked out from a clean branch. Claude works there with its native Edit and Write tools, which removes git apply and the round trip through diff text. It behaves like a human engineer: new branch, new checkout, then edit, commit and push.

Once the MR is merged or closed, the worktree is removed. A partial failure affects that worktree only, leaving the main checkout and other concurrent cases alone.

That is what makes concurrent agents safe on one machine at all. With no per-case worktrees, two cases on the same repo would race over filesystem state. With them, the cases are isolated by construction.

The harness fixes its own pull requests

The harness's source sits in a private GitHub repo. When the gap-report, finalize and MR loop yields an improvement MR for the harness itself, that MR runs through its own CI of pytest, linters and a Python syntax check. A separate service, pr-ci-fixer, handles any failure there in the same way mr-monitor handles failures on ticket MRs.

The result is self-healing for the code written for tickets and for the code written about the harness. Suppose an MR adds a new retry budget and breaks an existing test, failing CI. pr-ci-fixer spots it, starts a Claude session in a worktree, patches the problem and pushes again, so the human reviewer finds a green MR ready to merge. The loop has closed on actual harness changes more than once.

Pointing the loop at the harness's own source is the best check on whether it deserves the name production. An agent that can keep its own pull requests green is probably fit to be trusted with other people's.

Five layers of tests for five failure classes

The harness has 4,656 test functions across 126 files. That count means little without a framework for what each test catches, because tests aren't equal.

The L0 to L4 framework assigns every test to one of five layers according to what it isolates:

• L0, pure boundary. Nothing is mocked and there's no filesystem or IO. It covers pure logic like gate functions, JSON extraction and the confidence-ceiling arithmetic, and whatever passes at L0 holds in any deployment environment.
• L1, file isolation. It uses a real tmpdir filesystem and real threads, with pathlib unmocked. Races between concurrent writers and atomicity bugs in state files show up here, which L0 can't see by design.
• L2, service isolation. Real subprocesses run against mocked external APIs (Slack, GitLab, Claude). Bugs in how components fit together, including dispatch routing, payload schemas and state-file transitions, surface at this level, since L1 only isolates filesystem behavior.
• L3, sandbox. A real git repo is created through subprocess, while Slack and Claude responses are faked. Whole workflow sequences run against a real git object model, such as task registration, implement-phase kickoff and pending confirmation.
• L4, integration matrix. Its six sub-categories are actor contracts (L4a), the admin-by-role matrix (L4b), state consistency across runs (L4c), chains between services (L4d), external actor roles (L4e), methodology gaps (L4f) and temporal sequencing (L4h and L4i).
• L4g, production replay. Thirteen tests reproduce specific production incidents as regression tests. They're written after a bug turns up in production, encoding the exact actor, context and state that triggered it, and they're kept permanently so it can't return.

The L4g tests cost the most to write and are worth the most to keep. They encode institutional memory about failure modes that stay invisible until they happen again.

A style guide that keeps output from reading as machine-written

Each MR the harness opens passes through one more filter before it's submitted, a style SOP written specifically so the output doesn't look AI-generated. The rules are blunt:

• No AI attribution at all. No Co-Authored-By: Claude, no "Generated with Claude Code" and no [bot] suffix on commit messages. The MR belongs to the team and not to the model.
• No fill-in-the-blank headings. Templates like "What this MR does / why we need it" are banned. The description is continuous prose with a concrete Changes: bullet list, written the way an engineer would write it by hand.
• No bold inline tags. A Root cause: label followed by a paragraph is out, and so are Fix: and Risk:. Tag-style bolding is the most reliable machine-written tell.
• Proof of work has to be reproducible from a command. "Verified locally" doesn't count. pytest tests/test_foo.py::test_bar -v with the real command and the relevant 3 lines of output does.

At the moment the MR description is composed, the SOP is injected into the patch-generation prompt. It lives apart from the prompt template because operators curate it: edit it whenever you like, and the next MR follows the new rules, with no daemon restart or code change. Even the gap loop can propose SOP edits, so meta-rules about how the agent writes MRs travel as MRs against the file that holds them.

This matters more than it sounds. Nothing destroys trust in an automated MR pipeline faster than the team noticing that every description reads as machine-made. Once that impression sets in, the merge bar rises, the agent's genuine signal gets buried and operators start wanting to rewrite descriptions by hand, which defeats the purpose. The style SOP keeps the descriptions blending in with human-written ones, and that keeps the loop working.

One ticket, one MR, start to finish

Here is what happens when a real ticket arrives. The times are approximate.

End to end, this loop usually finishes in 30 to 90 minutes of wall-clock time for a small or medium ticket. The operator's total active involvement is typically under 5 minutes: confirming the analysis, optionally steering ambiguous decisions and optionally clicking merge.

Investigations accumulate. After 30 cases there are 30 gap_report*.md files, each noting something the LLM thought could be better. A admin: gap-patterns run every two weeks groups them into recurring failure modes, and the following finalize pass turns the biggest clusters into MRs aimed at the harness repo, which means the harness improves itself. This is the self-improvement loop made tangible.

How it compares with other harnesses

Several other systems share the harness shape. The table shows where this one overlaps with them and where it diverges:

Where this harness goes unusually deep:

• Pre-commitment hypothesis lock. Claude commits to at least six candidate hypotheses before a single investigation iteration runs, which prevents after-the-fact rationalization. No commercial harness in the comparison ships an equivalent.
• A 19-function quality gate framework with mechanical confidence ceilings. Confidence is capped by a fixed formula that starts at 1.0 and takes off 0.08 per open question and 0.05 per unchecked source, so it's more than a self-reported figure. Per-dimension adversarial penalties sit on top of three gate families covering logical consistency, structural completeness and assertion ceilings. None of the commercial harnesses compared ships this.
• Self-improvement that modifies the harness's own codebase. Gap reports turn into a filtered PR against the harness's own code, prompts and SOP. The nearest academic counterpart, SICA (arXiv 2504), reports benchmark gains but stays offline research, and no production system ships autonomous changes to its own harness code.
• Project-scoped SOP personalization. Every project's SOP evolves from its own failure-mode counts, and within 20 to 30 cases it captures that domain's recurring investigation failures. The comparison turned up no other production harness that does this.
• Multi-agent orchestration with explicit hand-offs. The chain runs from oncall_run to case_followup, gap_analyzer, finalize_case and dev-agent. Every link has a defined hand-off schema and an isolated worktree, so work is divided among agents rather than loaded onto one.
• Cross-machine state sync via git-backed watermark injection. One Claude session resumes on a laptop or a remote VM with its full context intact, and nobody syncs anything manually.
• Built-in CI auto-fix and reviewer auto-reply loops. Devin, a commercial product, is the only one with comparable self-healing built in, while every other framework requires custom wiring.

Where it's deliberately thin:

• There's no formal evaluation harness. The LLM isn't scored on a benchmark, and production feedback serves as the eval.
• There's no dashboard, only Slack and logs.
• There's no cost meter. Token usage is tracked loosely and isn't surfaced on a dashboard, with cost bounded instead by the human-in-the-loop confirmation steps.

What's left, and the meta-loop

The harness is still evolving, with open items in autonomous-loop coverage, cost metering and a formal eval harness, and none of them blockers. It already does what it was built for: turning Slack signals into merged code, answering reviewer feedback overnight, self-healing CI and feeding its own improvements back into its next version.

The real lesson is the meta-loop. Each scar in the harness is there because a real ticket put weight on it. Building and operating are one loop moving at two speeds. Production signal flows back to the harness's own source, the agent drafts a fix, a human reviews and merges it, and another class of bug gets closer to being trimmed away.

The system isn't finished. A production harness that believes it's finished has stopped accumulating scars, which is another way of saying it's no longer running under real load.

Build the loop first and make it autonomous second. An autonomous system without a feedback path into its own substrate just ships the same mistakes faster and in larger numbers, which is why the order matters.