Self-Improving Agents Through Skills: Reuse, Selection, Repair and Verification

A coding agent that stumbles on the same problem twice is usually missing a lasting place to keep what it learned. The idea behind a plugin called whetstone is that the work an agent does today should make tomorrow's work better. A helper the agent wrote and tested ought to be there next time. A workflow that needs a correction should lead to a durable change someone can review. And before a result is handed over, the checks that the task is known to require should already have run.

Those are ordinary engineering habits, yet they tend to live in a person's head and get repeated at the end of every session. Whetstone packages them as four skills that operate on other skills: reuse proven code, improve how skills get selected, capture procedural lessons, and verify output before it goes back to the user.

The framing is "self-improving agents" in a deliberately modest sense. The agent helps improve its own instructions and tools, a human reviews the changes and decides what becomes permanent, the model itself is not retrained, and nothing runs unattended in the background to learn on its own.

An initial audit of 383 skill files gave the plugin a concrete testing ground. The more transferable part, though, is the design: where reusable work should live, how to pick the right intervention, and what evidence to demand before a change is kept. That reasoning applies with the plugin or with a version built around your own workflows.

The part of an agent you can actually improve

A skill is a folder. It contains a SKILL.md file of instructions and, optionally, scripts and reference documents. The open Agent Skills specification fixes a common layout, with a brief name and description heading the file. Anthropic's engineering team has described skills as a way to give general-purpose agents reusable expertise, and the appeal is that you can improve a procedure without rebuilding the runtime.

Two separate jobs are involved. The agent must choose the right skill, and the chosen skill must then guide the work well. Fixing one doesn't automatically fix the other.

Whetstone gives each of its four skills a single responsibility:

• dry-promotion preserves reusable code.
• skill-audit inspects the information used for skill selection.
• skill-repair proposes procedural changes.
• verify-before-return checks the output.

What makes the split work is that each operation changes a different thing. A helper may deserve saving even though nothing failed. A vague description may need attention while the procedure underneath works perfectly. Pushing every request through all four operations would tie unrelated work together, whereas separate entry points let you invoke only the part a task needs.

Every operation ends in something you can look at: a file on disk, an audit report, a suggested diff or evidence of verification. That eases debugging and gives a reviewer a concrete thing to approve. Help from the agent is welcome, but a convincing explanation by itself never makes a change permanent.

1. DRY: keep the work that already succeeded

DRY stands for "don't repeat yourself," and for an agent that includes working code it produced earlier. When each presentation request forces a rebuild of the renderer, it has to solve an old engineering problem before touching the new content.

The renderer is the reusable machinery, and the content is what varies. Anthropic's documentation makes a related point with a PDF skill that runs a prewritten extraction script: code does the stable mechanical part, while the model makes the task-specific decisions.

That's the job of dry-promotion. It copies a helpful helper into the skill's scripts/ folder and rewrites the instructions so the script is executed. Saving a file alone doesn't do it: while the skill still tells the agent to write a script, nothing points it at the saved one.

A character counter is a good example from a writing workflow. A subtitle has a length limit, and a small program can compute the count while the model concentrates on wording. The skill could carry an instruction along these lines:

Before returning a subtitle, run the stored character counter. When the result is over the limit, shorten the wording and count again. Never guess the number, and never rebuild the counter.

What this buys is a stable implementation that can be inspected and tested. It doesn't make the whole workflow deterministic, since the subtitle itself can still vary, and a program can faithfully repeat a wrong definition. The counter has to settle whether it counts Unicode code points or visible characters, and how it treats spaces and line breaks.

Prepare a helper before promoting it: remove paths tied to one session, make the inputs explicit, and try it on cases beyond the one that inspired it. Then run the workflow again and check that the saved file is the thing being called. A run that quietly builds another counter has defeated the point of reuse, even when the final numbers match.

The same principle covers templates and shared references. When several skills share an output contract, keep one authoritative copy and have the others refer to it. Don't force two genuinely different workflows through one helper just because their first examples look alike. Reuse should preserve a stable operation and not hide differences that matter.

2. Help the agent pick the right skill

With progressive disclosure, an agent begins with lightweight metadata and loads a skill's full instructions only once it picks that skill. The practical reason is that loading every procedure up front spends context on work that may never be requested.

For automatic selection, the name and description therefore act as routing information. The carefully written explanation halfway down the file may not even be available when the model decides whether to open it.

Descriptions aren't the only route. Claude Code also supports explicit invocation, and runtime configuration affects what's available. Still, the description is the selection signal a skill author controls.

Scanning one person's skill library, with same-name duplicates counted apart from the collisions, gave this summary:

383 skills scanned
  missing description           0
  no trigger language         306
  colliding pairs               3
  duplicate names              23
  skills with >=1 defect      306  (79%)

The 79% is a truncated 306 divided by 383, about 79.9%. Those descriptions failed a wording check, which is not the same as 306 observed failures to use a skill. The scanner only checks for literal trigger wording like "use when" or "when the user". A model can follow a description lacking those words, and a description with them can still be far too broad to help. The output is a list to investigate and not a measure of how broken the agent is.

Give neighbouring skills different jobs

Consider two real descriptions from two skills in the same plugin:

One of them presents itself as an expert on building retrieval-augmented generation systems with vector databases, embeddings and advanced retrieval strategies. The other presents itself as an expert on selecting, configuring and tuning vector databases such as Pinecone, Weaviate, Chroma, Qdrant and Milvus for semantic search and RAG.

Both read well, so which one fires for a user who needs help picking a vector store for a RAG app? A capable model may well favour the second, which is narrower, yet the first still lays claim to the same ground. Nothing says what each skill owns, so the model has to infer the boundary.

Rewrites along these lines would draw a firmer line between them. They haven't been benchmarked:

name: vector-database-selection
description: >-
  Compare vector databases against workload, filtering,
  deployment, and operating-cost requirements. Use when
  choosing or replacing a vector store. For retrieval-pipeline
  design with an already chosen store, use rag-pipeline-design.

name: rag-pipeline-design
description: >-
  Design ingestion, chunking, retrieval, reranking, and
  evaluation for a RAG application. Use when building or
  diagnosing a retrieval pipeline. For a standalone vector
  database selection, use vector-database-selection.

The first owns the choice of database, and the second owns the route from documents to useful context for an answer. Some vocabulary stays shared, since the work genuinely overlaps.

Draft the test requests before touching any wording. "Compare vector stores for a self-hosted document search app" should land on database selection. "Where should our embeddings live? Help me decide" means the same thing in different words. "We already have a vector store but retrieval misses relevant paragraphs" belongs to pipeline diagnosis.

Then add a request that neither skill should own, such as fixing the spacing of a search button on mobile. Without it, you've tested which of two skills wins and not whether the agent can decline both.

Try them in fresh sessions with an identical model, enabled skills and project context. Leave the intended skill's name out of any automatic-selection test, because naming it changes the experiment. Where the runtime exposes it, record which skill got loaded, and keep the answer or artefact produced. A trace showing the file that was opened is better evidence than the agent claiming it used the correct skill.

Compare results before and after, and hold some paraphrases out of the rewriting session so you don't tune only for your own examples. This small regression suite shows whether a change helped on requests that matter.

3. Turn experience into reusable instructions

Spotting a problem leaves a more useful question: where should the correction live?

skill-repair writes its proposals as diffs for a person to approve, against a target that's under version control. The aim is the smallest edit that fixes the observed failure, so a reviewer can read it and revert it if it breaks something else.

Imagine a report-writing skill that hands back a summary without consulting the source document. Telling it to "be more careful" gives the agent very little to act on. What fixes it depends on why the check was skipped:

• Source never required by the procedure: change the procedure.
• Source required but its location not given: supply that context or ask for the missing input.
• Source read but a figure copied wrongly: a different check is needed.

One catch-all warning pasted into every case would skip the diagnosis entirely.

Keep the failed request together with the proposed edit, describe the expected behaviour, and preserve the artefact that exposed the failure. Rerun the request in a fresh session, then try a nearby request that should still work. Otherwise it's easy to approve a change because it reads persuasively, which is a comfortable way to spend an afternoon without improving anything.

Scope matters as well. A preference about one report stays with that report. A rule for every report the skill writes goes into the skill. A fact that changes often may sit better in a reference the skill looks up. If every correction lands in the permanent instructions, a reusable procedure gradually becomes a log of one person's last session.

Successes deserve the same care. When a sequence works reliably, preserve the useful order, inputs and examples and not just warnings about what went wrong. The Agent Skills authoring guidance recommends grounding skills in real work and refining them through execution. A skill should hold the knowledge the next run lacks, not a transcript of everything that happened.

When the shared guide needs the repair

One person's library is their own accumulated clutter, audited by their own tool, so the same audit was pointed at a public plugin repository kept by a university research-software team. In the snapshot it held 76 skills across 12 plugins.

The flags clustered by plugin. Seven plugins were flagged entirely, and one scientific-Python plugin had four of five flagged. That clustering prompted a search for shared templates and conventions, and the contributing guide turned out to be the source. It asked for "a clear description of the agent and its purpose" and offered a placeholder template that says what the agent does. It never asks for when to use it. Thirty-six skills had followed the guide correctly, and the guide itself was the defect.

Two public issues were reportedly opened, one for the guide and one for the 36 descriptions, and the order matters. Update the convention, show a good example, and only then bring existing descriptions into line. Otherwise the next contributor follows the guide perfectly and reintroduces the same pattern.

A CI check can help, but start it as advisory. If the rule knows only a few trigger phrases, making it a hard failure invites contributors to paste those phrases until the build passes, which lifts the number without lifting the skill. The real acceptance test still needs representative requests, including ones the skill should ignore.

4. Verify before the handoff

verify-before-return equips a skill with acceptance criteria and a review loop. Writing a file to disk is only a midpoint, and the workflow still has to inspect the result using evidence suited to the task:

• A slide deck needs rendering, then a look for cropped text, unreadable labels or elements off the canvas.
• For a research report, open the sources and check that they support the claims.
• For code, run tests that exercise the required behaviour.

Any claim of being done should point at what was actually checked.

Define those criteria before generation. Then have the skill draft, inspect, fix failures and rerun the checks affected by the fixes. Cap the revision loop and require disclosure of anything still unresolved, because "keep improving" can otherwise become endless editing without a clearer result.

Subjective work calls for a different review. A simulated beginner can flag unexplained terms, and a sceptical reader can challenge the argument. Both are useful, but both are model-generated criticism and not feedback from real audience members, so weigh each suggestion against the piece's purpose rather than accepting every objection. Judgement on taste and strategy stays with whoever commissioned the work.

Running a check isn't the same as running the right one, and the auditor's own testing shows why.

What testing the auditor changed

Testing showed the auditor had to treat duplicate installs and overlapping descriptions as separate findings. Its first run reported 81 candidate pairs, many of them identical skills sitting in different cached plugin releases. Looking at those pairs led to splitting same-name duplicates out of the collision analysis, which cut the candidates to 3.

That's why the records behind a summary deserve inspection before they steer any change. Duplicate files need an inventory check, while competing descriptions need a routing test. Lumping them together points people at the wrong repair.

Separating duplicate names also revealed 23 names present at more than one scanned path. Cached copies don't prove that old versions were active, which requires checking the runtime's enabled plugins and resolved paths. Likewise, 383 counts scanned files and not active, unique skills, and excluding same-name comparisons doesn't deduplicate the whole inventory.

The score is only a wording heuristic: words shared by both descriptions, divided by the distinct content words across them, with 0.35 as the default threshold. An exclusion can look like overlap, and two differently worded descriptions of the same job can slip through. The frontmatter parser also supports just a subset of YAML. Any flag needs a human look before it turns into a repair.

After a repair passes, keep the failed case along with one the old version already handled correctly. Together they show whether a later change fixes one behaviour by breaking another. No library-wide improvement in task success has been measured for whetstone, and these checks are how it would be evaluated.

The packaging doesn't travel; the skills do

Making whetstone usable beyond Claude Code was a goal. Sharing the SKILL.md format gets part of the way. A skill may still rely on tools, permissions or file paths that another client doesn't provide, and each client packages plugins differently. The Agent Plugins standard, the Claude Code reference and the OpenCode documentation describe those layouts.

Whetstone ships both manifests over a single shared skills/ directory. The procedure is kept in one place and each client's integration is tested on its own. Portable instructions still don't make models interchangeable, so run the same representative task on another compatible runtime and check its tool calls, dependencies and output before assuming the behaviour is the same.

Trying it on one workflow

No big library is required. Choose one workflow, such as a helper the agent keeps rebuilding, two skills it mixes up, or an output you routinely finish manually. Keep one representative request and its result to compare against.

Next, pick the intervention that fits: reuse an existing helper, sharpen selection, record a procedural lesson, or add a missing acceptance check. Then compare the following run with the saved example. A wider inventory can wait, and an old cache shouldn't be deleted merely because a recursive scan turned up old files.

The project carries an MIT licence. For Claude Code, its README gives these install commands:

claude plugin marketplace add cdcore09/whetstone
claude plugin install whetstone@whetstone

Four entry points are exposed: /audit, /promote, /repair and /verify. Use whichever fits the workflow. On a first try, it's wise to have it examine the relevant files and propose changes before anything gets applied.

To study the auditor without installing anything, clone the repo, read skills/skill-audit/scripts/audit_skills.py. From the top of the repository, the command below runs it, and only Python's standard library is involved:

python3 skills/skill-audit/scripts/audit_skills.py ./skills

Swap ./skills for the directory you want to examine, starting with a small collection you recognise before scanning every plugin cache on your machine. --json gives a structured report, and --threshold 0.25 shows a broader set of overlapping descriptions.

The script only reads files and never rewrites skills. An exit status of 1 means it found flags, candidate collisions or duplicate names, and isn't automatically a Python crash. Its default roots also don't cover every client's install layout, so an explicit path makes the scope clearer and easier to reproduce. If the report includes private skill descriptions or filesystem paths, keep it local, and share counts and an anonymised example in place of an inventory of internal projects.

The aim is for useful work to accumulate: code that can be reused, procedures that can be inspected, and checks that travel with the task. That is a practical entry point to self-improving agents, and one that stays under version control. Anyone whose workflow needs a different implementation can reason from four questions: what to reuse, what to select, what to learn, and how to check the outcome.