Agent Skills Are Code: Scope, Disclose, Script and Test Them

A SKILL.md file reads like documentation, yet editing it changes how an agent behaves. That makes it closer to code than to prose, and it should be designed, scoped and tested accordingly.

Skills aren't documentation

The usual first reaction to building an LLM skill is to write more: explain the architecture, add examples, describe edge cases and include background so the model grasps why things work as they do. That reaction is usually mistaken.

A skill isn't a manual for a developer learning your system. It's context injected into an agent that's busy trying to get something done. That difference shapes the whole design. The useful question isn't how to explain the system to the model but what the agent needs to know to make its next decision correctly.

From that angle, several choices get easier:

• write instructions as procedures
• make activation conditions specific
• load information progressively
• move deterministic work into scripts
• test skills on real tasks

Procedural knowledge only

Modern coding agents already handle a surprising amount of general software engineering. They know Git and npm, and they understand Angular, React, TypeScript, SQL, REST APIs, testing frameworks and CI configuration. A skill that merely repeats this contributes almost nothing:

When implementing an Angular component, use TypeScript.
Write maintainable code.
Follow Angular best practices.
Add tests.

It burns context without supplying anything the model lacked. The skills worth having capture knowledge particular to your environment:

When modifying an Angular application in this repository:

1. Locate the closest project.json belonging to the affected application.
2. Check whether the project uses Vitest or Karma before generating tests.
3. For applications under apps/trading/, do not import directly from apps/shared/.
4. Run the affected project's test target rather than the workspace-wide test command.
5. Before changing a public interface in libs/order-entry/, search for consumers in
   apps/trading-terminal and apps/blotter.tp

Here the agent has learned something it couldn't reliably infer from Angular's public documentation, namely how work should proceed in one specific repository. That's what skills are good at.

The Agent Skills format organizes this as a directory holding a SKILL.md and optional scripts and resources. Metadata is available for discovery, and the full instructions and supporting files load only when needed. The file format isn't the key point. The key point is the line between general model capability and knowledge your environment has to provide.

Context is a budget

Everything given to an agent costs context. Tool descriptions, MCP tool schemas, repository instructions, conversation history and skill descriptions all consume it, and an activated skill's body consumes more. So the question isn't merely whether some information is useful, but whether it's useful for the task at hand.

Say these skills are installed:

angular-testing
angular-migration
jira-workflow
postgres-debugging
release-process
accessibility-review
storybook
incident-response
performance-profiling

When the job is fixing a failing Angular unit test, the agent has no reason to know the release process or PostgreSQL debugging steps. That's why discovery matters nearly as much as content. Compare two descriptions. This one is trivially easy to trigger:

---
name: testing
description: Helps run tests in our repository.
---

This one is far more precise:

---
name: angular-unit-testing
description: >
  Run and debug Angular unit tests in projects using Karma or Vitest.
  Use for Angular unit-test failures and test creation.
  Do not use for Playwright E2E tests, React projects, or backend tests.
---

The second description does more than say what the skill is. It draws a routing boundary that keeps irrelevant instructions out of the working context. A talk on skill design for LLM agents makes this point well: skill metadata isn't administrative decoration. Sloppy activation rules pull useful instructions into the wrong tasks, while rules that are too narrow keep the skill from being found at all.

Progressive disclosure beats a giant SKILL.md

A common failure is letting SKILL.md grow into a small internal wiki. It begins with twenty lines, then someone adds deployment steps, then troubleshooting, then examples, then configuration options. Three months on, the agent must read 1,500 lines to change a single file.

Structure it the way you'd structure software instead:

angular-workflow/
├── SKILL.md
├── testing.md
├── migrations.md
├── architecture.md
└── scripts/
    ├── affected-projects.ts
    └── verify-boundaries.ts

SKILL.md then holds only the routing logic:

# Angular repository workflow

When modifying an Angular project:
1. Identify the affected project.
2. Read `architecture.md` when changing dependencies between projects.
3. Read `testing.md` when creating or modifying tests.
4. Read `migrations.md` when changing Angular versions or framework APIs.
5. Run `scripts/verify-boundaries.ts` after changing imports between projects.

Detailed testing instructions enter the context only when a task involves testing. Anthropic describes this as progressive disclosure: the agent loads metadata first, reads the main body once the skill proves relevant, and opens referenced files later, only if needed. The aim isn't to hand over as much knowledge as possible. It's to deliver the right knowledge at the right moment.

Let scripts enforce what scripts can enforce

Another boundary helps: instructions are for decisions, and scripts are for deterministic work. Picture a skill that says this:

Find all applications under apps/.
Check their Angular versions.
Ignore archived applications.
Compare their versions with the root package.json.
Report applications that are behind.

An agent can carry that out, but only by repeatedly inspecting directories, interpreting files, deciding what counts as archived, tracking state and probably making many tool calls. When an operation can be expressed exactly, write it once:

node scripts/check-angular-versions.mjs

The skill then simply says when to run it and what to do with the output:

When checking Angular version consistency, run
`node scripts/check-angular-versions.mjs`.

If it reports mismatches, inspect only the projects listed in the output.

This is better in several ways. The script can be unit-tested and behaves the same every time. It needs fewer model decisions and often fewer tool calls. And when something fails, you can reproduce the failure without asking an LLM to repeat its reasoning.

None of this means replacing agents with scripts. It means choosing the right abstraction. Use the model for judgment, such as deciding which projects an architectural change is likely to affect. Use code for exact algorithms, such as listing which projects import a given package. The agent orchestrates both, and that mix beats encoding an entire workflow in English.

Tools and skills solve different problems

A tool gives an agent an action, while a skill teaches when and how to use actions. A command-line tool might expose a way to compute affected projects:

tr project affected --base main

The tool supplies the ability. The matching skill supplies the organization's knowledge:

Before modifying a shared library:
1. Run `tr project affected --base main`.
2. If either `trading-terminal` or `blotter` is affected, run their integration suites.
3. If more than five applications are affected, stop and inspect the dependency change
   before proceeding.1

The CLI has no idea why five affected applications should look suspicious, and the skill needn't know how dependency traversal works. That's a clean boundary. The same split applies when the capability is delivered by a shell command, a web API, an MCP server or any other interface: don't bury policy in low-level tools unless it belongs there, and don't write deterministic algorithms as prose unless you must.

More skills aren't automatically better

A tempting approach to agentic development is to collect useful-looking skills, install them all and assume the agent got more capable. The evidence suggests being far more selective.

SkillsBench covers 86 tasks spanning 11 domains. According to the first version of its paper, curated skills raised average pass rates by 16.2 percentage points, with big differences between domains. Software engineering improved by just 4.5 points, and 16 of 84 tasks did worse once skills were added. Self-generated skills, written by the models themselves, brought no average gain.

SWE-Skills-Bench looked at 49 public skills on real software engineering tasks. Thirty-nine produced no improvement in pass rate, and the average gain was just 1.2 points. A handful of highly specialized skills, seven in all, improved results by as much as 30 points. Another three hurt, by as many as 10 points, since their guidance targeted a different version than the project used.

That fits what the architecture would predict. A generic skill competes with what the model already knows, while a very specific one can provide precisely the missing fact:

When our generated API clients change, regenerate schemas before running typecheck.

That has value. This has much less:

Always think carefully before modifying APIs.

The rule of thumb: don't create a skill because a task exists. Create one because there's procedural knowledge the agent keeps missing.

Skills need tests

This may be the biggest conceptual shift. Because changing SKILL.md changes behavior, the consequences resemble code changes:

• Remove one instruction and an agent might skip a required verification.
• Broaden the description and the skill may activate during unrelated work.
• Add a large reference document and token use may rise without better results.
• Rewrite a procedure and success rates may fall.

So test it. For a skill governing Angular migrations, an evaluation set might include a task like:

Upgrade application A from Angular 20 to Angular 21.

with expectations of this kind:

Expected:
- correct migration command executed
- generated migration changes preserved
- application tests executed
- unrelated applications untouched

A second task:

Upgrade a shared library consumed by applications A and B.

with its own expectations:

Expected:
- consumers discovered
- both affected applications typechecked
- dependency constraints respected

And a third:

Modify a React application in the same monorepo.

whose expected outcome is the absence of activation:

Expected:
- Angular migration skill is NOT activated

That final case counts as much as any successful run, so test both directions: the skill should activate here and shouldn't there. A tool named SkillGrade builds on the idea. Define tasks, run the agent repeatedly, then score every run using exact checks, an LLM-applied rubric over the recorded session, or both together. You can apply the principle without it. The point is to treat a skill as something that can regress. Change a skill without being able to say whether the new version beats the old, and you're editing production logic untested.

Start from failures, not a skill marketplace

The easiest way to build useless skills is to ask what skills a coding agent should have. You'll get a list of testing, debugging, code review, documentation, architecture, Git workflows, accessibility, security and twenty other topics the model already understands reasonably well.

A better starting point is the agent's own history. Look for repeated failures. Perhaps the agent:

• runs the whole test suite in a huge monorepo instead of working out the affected project
• keeps violating a dependency boundary in the repository
• takes the technically valid migration path but misses an internal compatibility step
• edits generated files instead of the schemas that generate them
• spends five minutes rediscovering the same debugging procedure each time one service fails

Those are skill candidates. The lifecycle goes like this: observe a repeated failure, identify the missing procedural knowledge, write the smallest useful skill, move deterministic steps into scripts, create representative eval tasks, measure, then keep, improve or delete the skill. Improving loops back to rewriting it.

Mind the final option, because deleting is legitimate. A skill that doesn't improve behavior isn't harmless. It adds context, maintenance burden and one more instruction that may clash with something later.

The real abstraction is controlled context

People often describe skills as a way to extend agents. That holds, but it hides a more interesting engineering problem. Supplying information to an agent is easy. Deciding which information becomes relevant during which execution is the hard part.

A well-designed agent environment ends up looking less like one giant prompt and more like a software system with layers. Always present are general model knowledge and small global instructions. Pulled in only when a task needs them are the task-specific skill, conditionally loaded references, deterministic tools and runtime repository context. Each layer has its own job. Blur those responsibilities and context bloats until behavior is hard to reason about. Separate them and skills become pleasantly boring, which is good.

A good skill is no showpiece prompt. It's a compact bit of procedural knowledge that fires in the right situation, leans on deterministic tools where it can, and is checked against the failures it was written to stop.