Building End to End AI Agents with ADK Dev Skills
End-to-end AI agents with Google ADK dev skills: what the skills are, which ones exist, and how they cover build, test, CI/CD and observability for agents.
This article is published in English.
The following notes reconstruct a practical path around “Building End to End AI Agents with ADK Dev Skills”. Emphasis stays on contracts, checks, and drop-in code placeholders rather than motivational framing.
Introduction
When working through the Introduction stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.
What Are ADK Dev Skills?
When working through the What Are ADK Dev stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.
Available Skills
When working through the Available Skills stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.
Compatibility
When working through the Compatibility stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.
Prerequisites
When working through the Prerequisites stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node. When working through the Prerequisites stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.
# Required tools
- Node.js (for npx)
- Python 3.10+
- uv package manager (pip install uv)
- Google Cloud SDK (gcloud)
- An active GCP project with billing enabled
gcloud config set project YOUR_PROJECT_ID
gcloud auth application-default login
Step 1: Install ADK Dev Skills
The Step 1 Install ADK stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts.
npx skills add google/adk-docs/skills -y -g
npx skills list
Step 2: Define Your Requirements
The Step 2 Define Your stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts.
I need to build a Retail Returns multiagent system using ADK.
Requirements:
- Check order status by order ID
- Process refunds for delivered orders
- Multi-agent architecture with routing
- Production deployment to Agent Engine
Step 3: Generate Project Scaffolding
The Step 3 Generate Project stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts. The Step 3 Generate Project stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.
Use adk-scaffold to create this project with:
- Agent Engine deployment target
- GitHub Actions CI/CD
- No datastore needed (we'll use mock data)
retail-returns/
├── app/ # Our agent code
├── tests/ # Unit, integration, eval tests
├── .github/workflows/ # CI/CD pipelines
├── deployment/terraform/ # Infrastructure as code
├── notebooks/ # Jupyter notebooks
└── Makefile # Common development commands
Step 4: Implement Business Logic
For the Step 4 Implement Business stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness.
Add tools for:
1. check_order_status(order_id: str) -> dict
2. process_refund(order_id: str) -> dict
Use mock data with 3 sample orders. Refunds only work for delivered orders.
# Mock database
ORDERS = {
"12345": {"status": "shipped", "total": 45.99, "items": ["T-shirt"], "refunded": False},
"67890": {"status": "delivered", "total": 120.50, "items": ["Shoes", "Socks"], "refunded": False},
}
async def check_order_status(order_id: str) -> dict:
"""Checks order status and details."""
if order_id in ORDERS:
return {"status": "success", "order": ORDERS[order_id]}
return {"status": "error", "message": f"Order {order_id} not found."}
async def process_refund(order_id: str) -> dict:
"""Processes refund with validation."""
if order_id not in ORDERS:
return {"status": "error", "message": f"Order {order_id} not found."}
order = ORDERS[order_id]
if order["status"] != "delivered":
return {"status": "error", "message": "Order must be delivered"}
if order["refunded"]:
return {"status": "error", "message": "Already refunded"}
order["refunded"] = True
return {"status": "success", "transaction_id": f"REF-{order_id}-123"}
from google.adk.agents import Agent
from google.adk.apps import App
from google.adk.models import Gemini
def create_status_agent() -> Agent:
return Agent(
name="status_agent",
model=Gemini(model="gemini-3-flash-preview"),
instruction="Look up orders using check_order_status tool.",
tools=[check_order_status],
)
def create_refund_agent() -> Agent:
return Agent(
name="refund_agent",
model=Gemini(model="gemini-3-flash-preview"),
instruction="Verify order is delivered before processing refund.",
tools=[check_order_status, process_refund],
)
root_agent = Agent(
name="triage_agent",
model=Gemini(model="gemini-3-flash-preview"),
instruction="Route requests to status_agent or refund_agent.",
sub_agents=[create_status_agent(), create_refund_agent()],
)
app = App(name="retail_returns_app", root_agent=root_agent)
Step 5: Test Locally
For the Step 5 Test Locally stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness.
Let's test this locally
cd retail-returns
make install
make playground
Step 6: Create Evaluations
For the Step 6 Create Evaluations stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness. For the Step 6 Create Evaluations stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.
Create an evaluation to test invalid order handling
{
"eval_set_id": "invalid_orders",
"eval_cases": [
{
"eval_id": "invalid_status_check",
"conversation": [
{"user_content": {"parts": [{"text": "What's the status of order #99999?"}]}}
],
"expected_tool_calls": [
{"tool_name": "check_order_status", "args": {"order_id": "99999"}}
]
},
{
"eval_id": "invalid_refund",
"conversation": [
{"user_content": {"parts": [{"text": "I want a refund for order #88888"}]}}
]
}
]
}
{
"criteria": {
"tool_trajectory_avg_score": {
"threshold": 1.0,
"match_type": "IN_ORDER"
},
"final_response_match_v2": {
"threshold": 0.8,
"judgeModelOptions": {"judgeModel": "gemini-3-flash-preview"}
}
}
}
make eval
Tests passed: 2
Tests failed: 0
Step 7: Deploy to Production
When working through the Step 7 Deploy to stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.
Deploy this to Agent Engine
# Export dependencies
uv export --no-hashes > app/app_utils/.requirements.txt
# Deploy to Vertex AI
uv run -m app.app_utils.deploy \
--source-packages=./app \
--entrypoint-module=app.agent_engine_app \
--entrypoint-object=agent_engine \
--requirements-file=app/app_utils/.requirements.txt
import vertexai
client = vertexai.Client(project="YOUR_PROJECT", location="us-central1")
agent = client.agent_engines.get(name="ENGINE_ID")
async for event in agent.async_stream_query(
message="Where is order #12345?",
user_id="test"
):
print(event)
Step 8: Set Up CI/CD
When working through the Step 8 Set Up stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.
Set up CI/CD with GitHub Actions
1. PR Checks (.github/workflows/pr_checks.yaml)
When working through the 1 PR Checks github stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node. When working through the 1 PR Checks github stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.
2. Staging Deployment (.github/workflows/staging.yaml)
The 2 Staging Deployment github stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts.
3. Production Deployment (.github/workflows/deploy-to-prod.yaml)
The 3 Production Deployment github stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts.
uvx agent-starter-pack setup-cicd \
--staging-project YOUR_STAGING_PROJECT \
--prod-project YOUR_PROD_PROJECT \
--repository-name YOUR_REPO \
--repository-owner YOUR_GITHUB_USER \
--auto-approve
Step 9: Observability
The Step 9 Observability stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts. The Step 9 Observability stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.
How's observability configured?
Built-in (No Configuration Needed)
For the Built-in No Configuration Needed stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness.
Optional Integrations
For the Optional Integrations stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness.
# Add to app/agent.py
from google.adk.plugins import BigQueryAgentAnalyticsPlugin
app = App(
root_agent=root_agent,
plugins=[BigQueryAgentAnalyticsPlugin(dataset="agent_analytics")]
)
Step 10: Write Tests for Production
For the Step 10 Write Tests stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness. For the Step 10 Write Tests stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.
Create integration tests that hit the deployed agent
import pytest
import requests
import subprocess
def get_access_token():
result = subprocess.run(
["gcloud", "auth", "print-access-token"],
capture_output=True, text=True
)
return result.stdout.strip()
@pytest.mark.integration
def test_valid_order_status():
"""Test order lookup on deployed agent."""
token = get_access_token()
response = requests.post(
f"https://us-central1-aiplatform.googleapis.com/v1/projects/{PROJECT_NUM}/locations/us-central1/reasoningEngines/{ENGINE_ID}:streamQuery",
headers={"Authorization": f"Bearer {token}"},
json={"input": {"message": "Where is order #12345?", "user_id": "test"}}
)
assert response.status_code == 200
text = "".join(event.get("text", "") for event in response.json())
assert "shipped" in text.lower()
assert "45.99" in text
pytest tests/integration/test_remote_agent.py
Before and After
When working through the Before and After stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.
Without Skills
When working through the Without Skills stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.
With Skills
When working through the With Skills stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node. When working through the With Skills stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.
Best Practices and Tips
The Best Practices and Tips stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts.
1. Install Skills Globally
The 1 Install Skills Globally stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts.
npx skills add google/adk-docs/skills -y -g
2. Be Specific in Prompts
The 2 Be Specific in stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Budget tokens per turn and per session. Agentic tools expand context aggressively; hard caps keep demos from becoming surprise invoices.
3. Leverage the Right Skill for Each Phase
4. Iterate with Evaluations
5. Review Generated Code
6. Use Version Control
git init
git add .
git commit -m "Initial scaffold from ADK Dev Skills"
Common Questions
Do we need to learn ADK first?
Can we modify the generated code?
Will this work with different AI assistants?
What if we want to deploy to Cloud Run instead?
Can we create our own skills?
How do we update the skills?
npx skills add google/adk-docs/skills -y -g