Teaching Deep Agents New Skills: Planning, Subagents and SKILL.md Files

Agentic AI is about systems made of several cooperating agents that split up a long, tangled problem, trade information and work like a small team of specialists. A lone chatbot answers one question. An agentic system needs planning, memory and tool use so it can carry a task across many steps. Reinforcement learning (RL) is the classic route to agents that learn from trial and error: a deep network watches an environment, picks actions such as tool calls or code execution, and gets rewards that steer its policy. Deep RL has produced superhuman play in Go and chess and strong results in robotics, all by maximizing expected cumulative reward. Bringing those ideas to LLM-powered agents raises an interesting possibility, namely agents that pick up reusable routines, or skills, and then plan and execute with less effort.

What makes an agent "deep"

Deep agents extend an LLM assistant with four capabilities:

1. A planning tool that splits complex work into steps.
2. Subagents for parallel work and context isolation.
3. A persistent file system that doubles as memory.
4. A detailed system prompt carrying goals and constraints.

Together these let an agent plan, keep state and run complicated workflows without choking its prompt context. A shallow chatbot answers once. A deep agent loops: it plans, searches, saves findings, synthesizes and refines. Asked for a report, it might write a todo list, run web searches or analysis, store notes in files, then draft and revise, all driven by the model with no human in the loop.

Reinforcement learning and where skills come from

RL is the foundation for agents that learn by acting. The agent observes a state, takes an action and receives a reward or a penalty, and by maximizing the expected sum of future rewards it improves its policy. It differs from supervised learning, which has labels, and from unsupervised learning, which hunts for hidden structure, because it deals with sequential decisions under uncertainty. Deep RL swaps in neural networks as function approximators so images, text and raw sensor data can be handled. Well-known milestones include Deep Q-Networks for Atari and policy-gradient methods such as PPO and A2C for continuous control and strategy games. Newer work looks at hierarchical skill learning, where agents discover options, meaning temporally extended actions that solve subgoals, which helps exploration and transfer.

For LLM agents, RL plays two roles. One is fine-tuning the model itself with feedback signals. RLHF, for instance, has aligned models to be safer and more on topic. DeepSeek-R1 showed that pure RL with no labeled data can train a reasoning model to rival top systems on math and code: the model generates several answers, an automatic scorer rewards correct ones and penalizes the rest, and over many rounds an algorithm like PPO improves the policy. Real pipelines may add supervised fine-tuning and rejection sampling for readability and safety.

The other role sits at the agent framework level, where RL-inspired methods can help acquire skills. An agent could record trajectories of what it did (which tools ran, what came back) and feed them to offline RL or imitation learning to pull out repeatable procedures. Looking at past wins and failures, it might compress frequently repeated tool sequences into higher-level routines. Those routines, written down as script files or instruction markdown, can be revealed gradually to the agent on later tasks. The system gains a form of continual learning: faced with something new, it learns a skill on the fly and stores it as a module.

How the DeepAgents library is built

LangChain's DeepAgents library turns these ideas into a production-minded harness, with an architecture shaped by multi-step agents like Claude Code and Manus.

• Planning with write_todos. A built-in tool has the agent draft a step-by-step list before acting. The list guides the work and can change as new information comes in.
• Filesystem and memory. File tools (ls, read_file, write_file, edit_file) let the agent push context out into files or databases. Storing intermediate output and notes prevents context-window overflow and lets it revisit earlier steps. A pluggable backend, whether in-memory, local disk, cloud storage or an isolated sandbox, handles persistence and safe code execution.
• Subagents. A built-in task tool spawns specialized helpers, say a research subagent or a critique subagent. Each runs in its own context, which keeps the main agent uncluttered, and can have its own system prompt and tools. The main agent may hand off web searches or code analysis and get back summaries. A general-purpose subagent simply mirrors the main agent's setup when isolation is all that's wanted.
• Skills. A skill is a folder holding a SKILL.md and, optionally, scripts or documents. Skills differ from tools because they aren't fully loaded into the prompt. Only the YAML frontmatter (name, description, tags) enters the context, and the full instructions are revealed progressively when the agent decides they're relevant. That cuts token usage sharply. A PDF skill might include a long tutorial that the agent reads only when the user's task involves PDFs. Skills spread domain knowledge and procedures across agents, and agents can even write new skill definitions as they meet new tasks.

Because of this design, deep agents do far better than shallow ones on long multi-step problems. They plan ahead, use memory, learn from experience and manage large contexts. Industry reports note that agent tasks are getting more complex, with length roughly doubling every few months, which is exactly where these features matter. The shift is from answering isolated queries to solving problems autonomously.

A step-by-step build

The plan below combines agent architecture with, where it applies, RL ideas. It uses Python and the DeepAgents SDK. Begin by installing the libraries and setting up the environment:

# Install deepagents and necessary tools
!pip install --upgrade deepagents langchain tavily-python

# Import libraries and set API keys (e.g., for LLM and search tool)
import os
from getpass import getpass
os.environ["OPENAI_API_KEY"] = getpass("OpenAI API Key: ")
os.environ["TAVILY_API_KEY"] = getpass("Tavily API Key: ")

from deepagents import create_deep_agent
from deepagents.backends import FilesystemBackend
from deepagents.middleware.filesystem import FileData
from langchain.chat_models import init_chat_model
from tavily import TavilyClient

Defining the agent and its skills

Next comes the deep agent with its tools and any predefined skills. For production a durable backend, such as local disk or LangGraph, is a better choice than ephemeral memory. This version uses a FilesystemBackend rooted at the current directory and loads skill folders:

from deepagents import create_deep_agent

# Assume we have local skill directories under "./skills"
skill_dirs = ["./skills/pdf_processing", "./skills/data_analysis"]

agent = create_deep_agent(
    model=model,
    system_prompt="""
You are a highly capable AI assistant. Your objectives:
1. Break tasks into steps using write_todos().
2. Use internet_search and file tools to gather and store information.
3. When given a task, plan and execute it step-by-step, writing to files as needed.
4. Load skills from the filesystem to handle specialized tasks when relevant.
5. Summarize results and refine final output before returning.
""",
    tools=[internet_search],             # Web search tool
    backend=FilesystemBackend(root_dir="./workspace"),  # Persistent file storage
    skills=skill_dirs,                  # Load skills from local directories
)

The system prompt emphasizes planning and skill use. The skills parameter points at directories whose subfolders each contain a SKILL.md plus supporting files. ./skills/pdf_processing/SKILL.md, for example, could teach the agent how to parse PDFs. With skills loaded this way, the agent initially sees only names and descriptions. When a user's task involves PDFs, it reads the relevant SKILL.md and any referenced files to learn the procedure.

An example layout:

skills/
├── pdf_processing/
│   └── SKILL.md
└── data_analysis/
    ├── SKILL.md
    └── analysis_script.py

And a possible pdf_processing/SKILL.md:

---
name: pdf-processing
description: Skill to extract and analyze content from PDF documents.
---
# PDF Processing Skill

To use this skill, follow these steps:
1. Use `pdf-tools` to read PDF content.
2. Summarize key findings from the PDF text.

...

The full instructions get read only if the agent decides the current request concerns PDFs. That's progressive disclosure at work: big skill files don't inflate every prompt.

What a run looks like

Now give the agent a complex task. The initial prompt nudges it to build a plan first.

from langchain.schema import HumanMessage

query = "Organize the latest quarterly sales data and write a summary report."
response = agent.invoke({
    "messages": [HumanMessage(content=query)]
})
print(response["messages"][-1]["content"])

Conceptually, the agent will likely do the following:

1. Call write_todos() to produce a checklist, such as load the sales file, make charts, compute totals, write the report.
2. Make tool calls, using internet_search or built-in file reads to collect existing data.
3. Possibly hand the cleaning or analysis to a subagent when the job calls for a specialized procedure.
4. Use skills. If a data-analysis skill exists, it might read SKILL.md from ./skills/data_analysis and follow it.
5. Save intermediate artifacts such as cleaned datasets or charts to the file system.
6. Write and polish the final summary.

A full run calls remote LLMs, so verification here is mostly about getting the structure right. The code picks a dependable backend, wraps risky steps in try/except (a full version should catch API and file errors) and can be extended with logging middleware. Unit tests for each tool, with the LLM responses mocked, would validate it.

Testing and evaluating

Agent workflows are evaluated mostly by judgment. The check is whether the agent finishes tasks correctly and sticks to its plan. One approach uses synthetic tasks with known answers: give it a small math project and compare the final report with the solution. Automated metrics help too, such as task success rate (did each step complete) or similarity to a reference answer scored by an LLM judge. Free-form text may still need human review or model-based critique for the final call.

If RL is used for skill learning, progress can be measured with reward curves. Set up an environment where finishing subtasks earns reward and plot total reward per episode, perhaps using OpenAI Gym or a custom setup. The metrics that matter are sample efficiency (how fast learning converges), final performance and stability across random seeds. In practice you'd also run a suite of tasks across research, coding and analysis, tracking success rate and token use for each.

Observability is essential in production. Instrument the agent with tracing and logging, since LangSmith can capture every tool call and LLM response. In a deployed system, logs can flow to CloudWatch or Prometheus with step counts, latency per call and errors. For RL components, log reward history, loss curves and hyperparameters. Stress tests with large batches, long sessions and odd inputs help confirm stability.

Deploying it as a service

In a real deployment the agent runs as a microservice. A scalable layout looks like this:

+------------------+      +--------------+
|  Client/User     | <--> | API Gateway  | <---> [HTTP requests]
+------------------+      +------+-------+
                              |
                              v
                       +-------------+
                       |  AgentCore  |   <-- AWS Bedrock AgentCore (managed runtime)
                       +-------------+
                              |
                +-------------+-------------+
                |                           |
      +-------------------+       +-------------------+
      | Deep Agent Service |       | Storage (S3/DB)   |  <-- File/memory backend, logs
      +-------------------+       +-------------------+
                |                           |
                +-------------+-------------+
                              |
                      +---------------+
                      |   LLM Models  |  <-- e.g., OpenAI, Anthropic, local LLMs
                      +---------------+

An agent service on AWS AgentCore or something similar receives prompts, coordinates subagents and makes tool calls, with every interaction recorded in a persistent store for observability and memory. A few design points stand out.

• Scalability and reliability. Run the agent in a serverless or containerized runtime such as AWS Bedrock AgentCore, Azure AI Agent Service or Kubernetes. It scales with load and starts fresh instances per request, and each session can sit in its own micro-VM. A persistent backend (S3, a database or the LangGraph store) keeps state and memory across restarts.
• Observability. Trace every invocation. On AWS, the Bedrock AgentCore starter kit wires up CloudWatch and X-Ray for latency and failure monitoring. Logging hashes of prompts and responses instead of raw sensitive data still lets you track performance, along with token usage, API call counts and success rates.
• Security. Limit what the agent can touch. Filesystem tools should work only inside designated directories, and code execution under the sandbox backend should happen in hardened environments such as Modal or Daytona. Give any cloud resources the agent calls least-privilege IAM roles, and add input validation and content filtering through LLM safety checks against malicious prompts.
• Cost. Balance model and backend choices. A cheaper Anthropic model can take routine work while a premium model like GPT-4o is kept for the planning steps that count. Offloading context to memory and reusing tool results cuts repeated prompt tokens.

A complete example

This Python sample implements the plan, with error handling and comments. It's conceptual and needs real API keys and an environment to run.

from deepagents import create_deep_agent
from deepagents.backends import FilesystemBackend
from deepagents.middleware.filesystem import FileData
from langchain.chat_models import init_chat_model
from tavily import TavilyClient
import os

def setup_agent():
    """Initialize the Deep Agent with tools, skills, and system prompt."""
    # Model and tools initialization
    model = init_chat_model(model="openai:gpt-4o")
    tavily_client = TavilyClient(api_key=os.environ["TAVILY_API_KEY"])
    def web_search(query: str, max_results: int = 3) -> str:
        results = tavily_client.search(query, max_results=max_results)
        return "\n".join(f"{r.title}: {r.summary}" for r in results)

    # Create the deep agent with planning, files, and skills
    try:
        agent = create_deep_agent(
            model=model,
            tools=[web_search],
            system_prompt="""You are an expert agent. Your workflow:
            1. Create a plan with write_todos().
            2. Use tools (e.g. web_search) and file system to research and work.
            3. If specialized tasks arise, load relevant skills from the 'skills' directory.
            4. Save results and refine the output at each step.
            """,
            backend=FilesystemBackend(root_dir="./workspace"),
            skills=["./skills/pdf_processing", "./skills/data_analysis"]
        )
        return agent
    except Exception as e:
        print(f"Error initializing agent: {e}")
        raise

def invoke_agent(agent, user_query):
    """Invoke the agent on a user query and return the final answer."""
    try:
        messages = [{"role": "user", "content": user_query}]
        result = agent.invoke({"messages": messages})
        return result["messages"][-1]["content"]
    except Exception as e:
        print(f"Agent invocation failed: {e}")
        return None

# Example usage
if __name__ == "__main__":
    agent = setup_agent()
    task = "Analyze the recent research on climate change and summarize key findings."
    answer = invoke_agent(agent, task)
    print("Agent response:", answer)

Agent creation and invocation sit in functions with docstrings and try/except blocks. The web_search tool uses Tavily for outside data, and the system prompt tells the agent to plan, use skills and refine its output. A production version would check edge cases like empty search results and API timeouts and would log exception details instead of printing them. Secrets belong in a secure environment, not in the source. The structure extends naturally with more tools or subagents.

To validate it properly, write unit tests. Mock the LLM to return fixed planner output and check that the agent writes the expected files or calls tools in the right order. Measure inference latency and token cost on benchmark tasks as well.

Trade-offs to plan for

Deep agents join planning, memory, subagent delegation and loadable skills to handle multi-step jobs that simple chatbots can't. A clear system prompt, a planning-first workflow and a rich toolbox give a sturdy base for autonomous assistants. Skills hold modular knowledge, for example pdf-processing or deploy-to-prod, loaded on demand so context stays lean and the agent can keep learning.

• Correctness against cost. Strong models such as GPT-4o or Claude Sonnet plus extensive searches improve accuracy and raise token and API bills. Mixing model sizes and caching helps balance quality and expense.
• Scalability against complexity. A serverless agent runtime like AWS AgentCore brings autoscaling and observability, but there's integration work: containerizing the agent code and configuring networking.
• Security against flexibility. Sandboxed execution through Modal or similar blocks code injection risks, though it restricts operations such as arbitrary network calls. Define clear tool interfaces for what's needed.
• Reliability. Retry transient errors like LLM timeouts, and consider human checkpoints on critical steps, since some agent frameworks support interrupts.
• Observability. LangSmith or custom logging can trace every planning step and tool call. Watch CPU, GPU and latency, and keep checking output quality.

Looking ahead, skill acquisition itself could be automated: RL or unsupervised learning over past trajectories might generate new skill definitions. Multi-agent frameworks such as Microsoft AutoGen or Salesforce AgentForce could coordinate several specialist agents talking over LangGraph protocols.

Building an agent that learns skills takes two things: the right software stack and, if desired, RL techniques that improve it over time. With DeepAgents and disciplined engineering, you can ship assistants that solve complex problems on their own, learn new procedures and run reliably at scale.