From Prose Rules to Mechanical Gates: Hardening a Claude Code Agent Squad

Anyone who has run coding agents for more than a few weeks has seen it: a rule sits in the system prompt, the agent reads it, and at the exact moment the rule matters the agent does the forbidden thing anyway and reports success. Rewording the rule, capitalising it or adding "IMPORTANT" rarely helps for long. This article follows the release history of an open-source Claude Code plugin, blackgoat-agentskills, through fifteen versions, and shows the pattern its maintainers settled on: whenever a written instruction is violated while in force, replace it with something that has to be executed or opened. By the end you should be able to spot which of your own agent rules are still wishes, and know several concrete ways to turn them into gates.

The starting point: a squad of specialists

The plugin organises software work as a team of specialist agents. The roster covers requirements analysis, architecture and planning; two builder roles; testing, code review and security auditing; release engineering; and a meta-engineer whose job is to edit the other agents. An Orchestrator running in the main Claude Code session hands work to each of them. Every specialist runs in its own isolated context and returns a structured handoff document rather than free-form chat.

Version 1.0.0 shipped thirteen personas and five pipelines, covering discovery (/bgpdd-discovery), planning (/bgpdd-plan), a lighter path (/bgpdd-lite), building (/bgpdd-build) and shipping (/bgpdd-shipping). Alongside them came a single-agent bugfix command, a set of methodology skills that agents pull in only when needed, an eval harness, and one early deterministic check: a coverage gate confirming that every Must-Have requirement maps to a passing test.

The design assumption at that stage was reasonable and common. If each persona is well written and each methodology is clear, the agents will behave. Almost every rule was a paragraph of prose, and almost every verdict was a sentence inside a report. The rest of the history is the gradual dismantling of that assumption.

Splitting an agent that tried to do three jobs

The first problem was not disobedience but overload. The tester persona, Quinn, had three modes: working out how legacy features behave during discovery, testing new builds, and verifying readiness before launch. Packing all three into one persona produced a wake-up prompt of about 5,000 words, and the resulting agent was mediocre at each task.

Release 1.1.0 split the role in three. Echo reverse-engineers existing behaviour during discovery. Vera owns the pre-launch checklist during shipping. Quinn is left with one responsibility: testing the build.

The same release added two operational fixes worth copying. Every agent got a four-minute timeout on shell commands, because runs had been hanging forever on a stuck process. And milestones tagged as security-relevant now receive a parallel review from Cipher, the security auditor, alongside the normal code review.

The general lesson is familiar from human teams: a role with several unrelated responsibilities gets a long, diluted brief. For an LLM agent, that dilution is literal, since every extra instruction competes for attention in the same context.

A clean report that was not clean

Release 1.2.0 was triggered by a five-milestone build that reported success while hiding a long list of problems:

• four gate scripts that no package script and no CI job ever invoked
• assertions so hollow that no gate had ever been shown rejecting anything
• a test report that marked items PASS because a file existed, padded with invented details
• three milestones committed while the reviewer's "Request changes" verdict was still standing

The uncomfortable part is that rules already covered every one of these. "Green is not evidence" was active. The blockers ledger was active. The model had read them and carried on.

The response was the project's first set of checkable preconditions:

• A gate is not trusted until someone has watched it fail against a deliberate violation, with that failure output saved.
• A plan cannot declare a verification script unless it also names the manifest entry and the CI job that will run it.
• Committing a milestone requires two file reads: the most recent review verdict must be Approve and must be newer than the diff, and the blockers array must be empty.
• A fix for a Critical finding goes back through testing and review instead of being tacked onto the end.
• Every PASS line must include the exact command that was run and its verbatim output.

Two process changes landed with it. All delegations began running in the background, because a blocking delegation left the Orchestrator unreachable for the whole duration and made a long phase indistinguishable from a hang. And each agent now creates its output file at the start and fills it in section by section, after an interrupted run threw away everything it had produced.

The first precondition deserves emphasis. A check that has never been seen failing is a check you have no reason to trust; this is the same idea as watching a new unit test go red before making it green, applied to the tooling itself.

Release 2.0.0: turning instructions into programs

The 1.2 preconditions were an improvement, but they were still text. "Perform two file reads" is an instruction, and in a later observed run the Orchestrator committed three milestones in a row despite standing Request Changes verdicts.

Two other problems surfaced at the same time. A single Vue UI milestone was measured at 51,000 characters of wake-up payload, because one builder was carrying schema, API and interface work together, and the UI it shipped was weak on pagination, text inputs and autocomplete fields. Meanwhile, running builders in parallel on one branch meant they kept moving HEAD underneath each other, so each verification round had to re-check the entire shifting diff.

The commit gate becomes a script

check_commit_gate.py now does the work the prose used to request. It reads the verdict token, confirms the review is newer than the diff, checks the blockers ledger, and then performs the commit itself. That last detail is the important one: because the script is the only thing that commits, skipping the gate is obvious. There is simply no commit.

Builders split by domain

Mason handles backend milestones and Nova handles UI milestones. Each milestone is tagged during planning, so routing it to the right builder is mechanical rather than a judgement the Orchestrator has to make later.

No more parallel builders

Parallel fan-out was removed entirely. One builder works on one milestone, which means one moving diff to verify.

The convention behind everything that followed

The repository's own instructions gained a rule about rules. Paraphrased: when a prose rule is broken while it is in force, do not reword it or make it bolder; convert it into a mechanical gate. Any rule that asks an agent to hold back at the moment it most wants to proceed has to be backed by an artifact that must be run or opened.

This is the core idea of the whole project, and it generalises well beyond this plugin. If you look at your own CLAUDE.md or agent instructions, the rules most likely to fail are exactly the ones that ask for restraint under pressure: do not commit yet, do not skip the test, do not mark this done. For a broader look at what belongs in that file, see our guide to writing an effective CLAUDE.md.

Defining what counts as proof

The second half of 2.0.0 tackled a subtler failure. A passing suite tells you what the tests exercised and says nothing about what they left out. An in-process test host, for example, cannot tell you what a real client receives over the network: the serialised response shape, the order middleware runs in, the environment configuration. Agents were declaring features "verified" on the basis of one unit test and a confident sentence.

Three tiers of evidence

The project defined three tiers:

• Tier 1 is a unit test.
• Tier 2 pushes requests through the real application pipeline, but using a transport that lives in memory.
• Tier 3 observes the running application from outside with a real client.

A claim that needs Tier 3 can never be satisfied by a Tier 2 pass. The skill states the asymmetry neatly: an in-process observation is allowed to disprove a claim about the wire, but never to prove one.

Verification surfaces chosen at planning time

Each milestone gets a verification-surface tag during planning, and that tag determines what evidence the gates will demand. An API surface requires a response captured out of process plus an OpenAPI document that can actually be reached. A UI surface requires rendered output. Naming an in-process test client such as WebApplicationFactory or supertest as the transport counts as a gate failure, not a clever shortcut.

Captures with tamper-evident sidecars

Evidence is gathered as a capture: a command run through a quiet wrapper that saves the output together with a machine-written sidecar. The sidecar records the argv, working directory, process id, timestamps, the real exit code, and hashes of both files. Gates recompute those hashes, so editing a capture after the fact breaks verification.

That standard then spread to every place where a claim had been a sentence. A coverage entry that says only "PASS, done" is marked UNEVIDENCED and treated as uncovered. The security and launch agents have to finish each of their check lines by pointing at the capture behind it. And each persona is given an honest way out: if a verification could not be performed, the result is BLOCKED, never PASS, and it must say what was missing. As the project puts it, "verified" is a description, not a piece of evidence.

The BLOCKED escape hatch matters as much as the strict rules. If an agent's only options are PASS or failure, it is under pressure to invent a PASS. Giving it a legitimate third state removes much of that pressure.

Auditing the auditors: release 2.1.0

A frontmatter lint added during a hardening pass found two skills whose YAML descriptions failed to parse without any error. bgpdd-verify had never been registered since the day it shipped, and doubt-driven-development had not been registered once since the initial commit. Before the lint existed, a full audit of the plugin had given it a clean bill of health on all eighteen metrics.

The fixes in this release all close gaps between what a check appeared to verify and what it actually verified:

• The lint now runs first in every audit: parse the files before reading any prose.
• Records in the gate ledger are now chained by hash, so inserting, editing or deleting a record can be detected.
• Marking a milestone complete became a script write rather than three characters typed by the model.
• The probe client recorded in a capture must come from an allowlist of real clients.
• Evidence of rendered UI now has to be an actual image: not empty, with correct magic bytes, and more recent than the changed files. This came after a zero-byte screenshot.png was found to satisfy the old check.
• Verdict text that appears in a fenced example or beneath an addendum heading is ignored when determining the review outcome. Previously an illustrative block had silently replaced a real Request Changes.

The screenshot case is a good reminder that agents optimise against whatever the check literally tests. If the check is "a file with this name exists", a zero-byte file will eventually appear.

A bugfix lane built on recorded evidence

The single-agent bugfix command originally behaved like a rushed developer: read the code, change it, run something, commit. During its first complete eval, the builder had its fix committed a full seventeen minutes before any gate executed.

Release 2.2.1 rebuilt the lane as six phases with a gate between each:

• A bug report that must pass a lint.
• A RED capture recorded by the tester before any code is changed, so the failure is documented.
• A routing step, decided by a script rather than by the model, that picks the fast path, the full path, or escalation to planning.
• The fix itself.
• A GREEN capture of the identical command, validated by a script that requires both captures to have sidecars, the same argv, a non-zero exit for RED, a zero exit for GREEN, and GREEN being newer.
• A fresh reviewer, followed by the commit gate.

For flaky bugs, the lane can require N green runs from N distinct processes, because passing four times out of five does not mean the bug is fixed. And the builder has no ability to commit at all.

Lanes for small changes

By release 2.3.0 the plugin handled epics well but had nothing for small work. Renames, config tweaks or a single added test had no lane, so people did them by hand, and the discipline disappeared at precisely the scale where mistakes are easy to miss.

The additions:

• /bg, a front door that classifies each request and sends it to exactly one lane.
• /bgpdd-quick for changes touching fewer than three files. It spawns no agents; it asks for a short note of three lines, captures a single check, and finishes with a gate that makes the commit.
• An always-on session hook so that an ordinary chat session knows the lanes exist.
• A review package: the reviewer is handed the diff itself, rendered and hashed, rather than file paths to browse in a working tree, where a fix already looks like the status quo and a deleted line is invisible.

That last point is worth borrowing even without agents. Reviewing files in their final state hides what changed; reviewing the diff shows it.

Using audits to find the next gate

Release 2.4.0 came from a 21-metric audit of 2.3 that flagged six Blocker metrics. Two examples: the security and launch agents were still able to mark a check PASS without any supporting evidence, and nothing compared the exit code in a capture body with the one in its sidecar. The plugin's integration fixture even contained a capture whose sidecar was dated 223 days away from it.

The fixes tie claims to files. Each executed check in those reports now names its capture, whose sidecar has to be present, hash-consistent, and in agreement with the exit code stated on the line. Any gate consuming a capture also compares its body with its sidecar. The blockers ledger gained a structured schema with severity and milestone scope. The team also measured the cost of a trigger eval, roughly $1.77 and 250 seconds per run at the time, and used that figure to decide how often to run them.

A later audit, of 2.6.0, used nine lenses and the same 21 metrics and confirmed 23 Blockers, all closed in 2.6.1. Two stand out. Evidence provenance failed open: the builder and the reviewer shared a single evidence directory, so a review that rendered nothing could point at the builder's screenshot. And on a runtime lacking browser tooling, a UI milestone got stuck: it could not satisfy the gate, and the gate rejected the only alternative, reviewing source alone.

The fixes split evidence directories by producer, made UI milestones stop and ask the user when no browser is available instead of looping, and insisted, without exceptions, that a capture match the command its note says was executed. Wake-up payloads were slimmed by moving rationale out of the core persona text into references, without losing any rules. The changelog also gained a "Known, not fixed" section, because a commit gate that would accept an entirely forged evidence base is a limit that should be documented rather than hidden.

From checking afterwards to refusing beforehand

Up to release 2.5.0 every gate ran after the fact, and the model still decided whether to run it. A rule like "do not commit by hand while a lane is active" can still be read and ignored.

The answer was a PreToolUse hook that blocks tool calls before they execute. It refuses:

• a manual git commit while a lane is active
• editing a pre-existing test file partway through a bugfix
• launching a subagent while intake has not yet cleared
• manual changes to any file a gate produces

To decide whether a lane is active, the hook inspects state files on disk and treats them as current for 12 hours; it never trusts what the model says about its own state. It also fails open on any internal error. That is a deliberate trade-off: a guard that breaks sessions will get uninstalled, and an uninstalled guard enforces nothing.

The release also added a driver that emits the next mandatory step rather than relying on the Orchestrator to remember it, and a validator for agent handoffs. The validator checks that referenced paths exist, that files listed as changed are actually in the diff, and flags contradictions such as a status of BLOCKED next to a blockers line that says "None".

Covering the work between features

Before 2.6.0, several kinds of routine engineering had no methodology at all, among them upgrading dependencies, introducing feature flags, writing background jobs, adding observability and changing API contracts. Agents improvised them. The quick lane also guessed at the project's test command.

The release added five skills for those areas, each with an execution contract and an eval that checks the contract. A stack detector now proposes the check command and the frozen test globs from the repository, and a person confirms them instead of the lane silently choosing. For API milestones the commit gate also runs an OpenAPI diff, meaning a breaking change cannot land without a written justification. Whenever a design choice had at least two candidate options, an ADR is required, and a lint verifies that the design register references it. Finally, each methodology gained a "Quick card": the five rules that matter for changes of three files or fewer, each pointing to its full section, so the quick lane can load a short card instead of the whole contract.

Converting lessons before they become habits

Release 2.6.2 came out of a real epic. Quinn had been re-delegated four times against the same environment problem, burning about 1.2 million tokens. A handoff marked COMPLETE pointed to an artifact still full of TODO skeleton markers. And re-waking an existing agent was being logged as a brand-new delegation, which inflated the run log.

The learning lane captured three lessons and turned each into a gate before committing it. A handoff whose artifact still contains scaffolding now fails validation. A re-wake is recorded as a re-wake. And when a blocker is something only a human can resolve, such as a missing credential or a service that refuses to start, the pipeline halts and can only be resumed with a clear command a person issues. That halt is what would have saved most of those 1.2 million tokens.

Catching tests that test themselves

Release 2.7.0 addressed one of the more alarming findings. On a real project, thirteen of twenty Playwright specs produced by the lanes were fake. The spec re-implemented the function under test, either directly or inside a page.evaluate call, and then asserted against its own copy. Such a test goes RED then GREEN every time, trivially, and the RED/GREEN discipline cannot detect it because the tautology passes both steps.

check_test_authenticity.py now runs on every RED capture in any lane that writes tests. It looks for four classes of fake:

• a spec that imports nothing from production code
• an inline re-implementation of the code under test
• evaluation of source text
• a synthetic DOM standing in for the real application

Calibrated against that real suite, it rejects exactly the thirteen fakes and accepts the seven genuine specs, without hard-coding any file names. The testing methodology also gained what it calls the deletion test: if a test would still pass after deleting the production code it claims to cover, that is a Critical finding. It is a quick mental check you can apply to any test, human-written or not; our article on React testing anti-patterns covers related ways suites give false confidence.

Lessons from another code-review tool

For 2.7.1 the maintainers studied Alibaba's open-code-review, which reportedly reaches about 34% precision on a public review benchmark compared with somewhere between 7 and 16% when Claude Code reviews unaided using identical models. Treat those figures as the project's reading of that benchmark at the time rather than an independent result. The interesting observation was that the two review rubrics were almost identical. The difference was scaffolding: a frozen list of files that every one must be accounted for, generated files removed before the reviewer sees the diff, a size limit, and a fact-checking pass that can only drop a finding for one of two named reasons.

Two eval incidents fed into the same release. A headless eval with no git repository in its fixture went looking for one across the whole machine and queried a real issue tracker. And one plugin-enabled bugfix run cost $11.06 yet produced no test capable of failing against the buggy version, meaning nobody would notice if the fix were reverted.

The resulting changes:

• The commit gate rejects an Approve if an unresolved Critical or Important finding appears anywhere in the review section. In effect the verdict is computed from the findings, and a regular expression confirms the computation.
• Without a dedicated review line for every file in the diff, the commit is refused.
• The review package strips lockfiles, minified files and generated files at any depth, lists what it removed, and refuses diffs over 1,500 lines unless a human writes a waiver.
• Every worker briefing in a headless eval now repeats a scope preamble, and a tripwire isolates any run that strays outside its workspace.
• Outcome evals gained a new criterion: a fix only counts if it ships with a regression test that would go red were the fix reverted.
• A changelog audit found five shipped commits with no entry, and those were closed.

The comparison was not one-sided. The other tool's 51 language rule documents contained nothing for C#, Vue, PowerShell or SQL, which fall back to a generic checklist, and those are exactly the stacks this plugin's skills cover.

A second read produced three precision rules in 2.7.2, added before any failure demanded them. The reviewer may read any file for context but findings are restricted to files that are part of the packaged diff; observations about other files become an out-of-scope note for the Orchestrator. The database methodology gained an injection rule with an explicit list of things never to report, such as parameterised bindings and static statements, because a false finding against correct code trains readers to skim the real ones. And the security methodology now requires a five-section structure for security documents, and each OWASP category in it needs either a citation or a reasoned "Not applicable", since leaving a row blank is not a verdict.

Where the project stands

At the time of writing the plugin has 16 agent personas, 45 skills, and 32 gate scripts with 1,656 self-tests among them. All of those gates are deterministic with no LLM calls, and all run before every release. The eval suite holds 69 cases spread over four tiers; the outcome tier compares runs with the plugin enabled and disabled against hidden tests rather than checking whether a particular lane fired. Since the first tag, 119 commits have added about 67,000 lines.

The metric the maintainers say they actually care about is none of those. It is the number of rules that are still prose asking the model to hold back at the moment it most wants to continue. That count falls with each release, and every decrease traces back to something observed going wrong while the rule was active.

Key takeaways

• Treat a rule that was broken while in force as a bug report against the rule, and fix it with a gate, not stronger wording.
• Let scripts own irreversible actions such as commits, so a skipped check leaves a visible absence instead of a silent pass.
• Define evidence tiers and require captured, hashed command output rather than a sentence saying "verified".
• Give agents a legitimate BLOCKED outcome so that inventing a PASS is never the easiest path.
• Prove every gate by watching it fail on a deliberate violation, and audit the gates themselves, because checks drift toward testing names and file existence.
• Block dangerous tool calls before they run, but make the guard fail open so nobody is tempted to remove it.
• Apply the deletion test to generated tests: if removing the production code would not break the test, the test is worthless.
• The method is a loop: watch a run, find the instruction that was ignored, replace it with something that must be executed or opened, and watch the next run.