Practical notes: Building an Agentic Playwright Harness for Enterprise UI

This walkthrough rebuilds the path from raw materials to a working system for: Building an Agentic Playwright Harness for Enterprise UI Automation. The focus is operable steps, explicit checks, and code that you can drop into a repo without guessing intent. For the Overview stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

The Problem We Were Solving

When working through the The Problem We Were stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours.

Use the harness to add manager role CRM coverage.

check MCP access,
inspect existing coverage,
use Playwright Agents or capture workflows,
reuse framework layers,
run focused tests,
update the graph,
and report evidence.

The Everyday Workflow

When working through the The Everyday Workflow stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours.

Prompt: create tests for manager role flow
  -> check whether MCP is available and allowed
  -> if yes, use official Playwright Agents for exploration or healing
  -> if no, use capture workflows and Playwright CLI evidence
  -> in both paths, follow the same harness standards
  -> verify with npx playwright test

Mode 1: MCP Available

When working through the Mode 1 MCP Available stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours. When working through the Mode 1 MCP Available stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

Official Playwright Agents
  + MCP browser context
  + Harness standards
  + Playwright test verification
  = faster generation with team governance

Mode 2: MCP Not Available

The Mode 2 MCP Not stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve.

Prompt
  + local context files
  + capture workflows
  + playwright-cli snapshots
  + npx playwright test
  + trace/report evidence
  = no-MCP agentic Playwright workflow

Capture Workflows: The Practical CLI Layer

The Capture Workflows The Practical stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve.

playwright-cli open http://127.0.0.1:4173/login
playwright-cli localstorage-set ...
playwright-cli goto http://127.0.0.1:4173/crm/accounts
playwright-cli snapshot --filename=contexts/captures/crm-manager-accounts-loaded.yml

npm run capture:recipes:generate
npm run capture:recipe -- --list
npm run capture:crm:manager -- --dry-run
npm run capture:crm:manager

Day 1, Day 2, And Later

The Day 1 Day 2 stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve. The Day 1 Day 2 stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

Day 1: Add A New Flow

For the Day 1 Add A stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Authenticate at the gateway and re-authorize at the data plane. A bearer token alone is not a tenancy boundary.

Create manager role flow tests and assert which buttons are accessible and not
accessible.

Day 2: Run The Existing Tests

For the Day 2 Run The stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Authenticate at the gateway and re-authorize at the data plane. A bearer token alone is not a tenancy boundary.

npm test
npm run test:smoke
APP_BASE_URL=http://127.0.0.1:4173 npm run sample:crm:test

Later: Refresh Only What Changed

For the Later Refresh Only What stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Authenticate at the gateway and re-authorize at the data plane. A bearer token alone is not a tenancy boundary. For the Later Refresh Only What stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

Failure on manager reports page
  -> inspect Playwright output and trace
  -> check existing reports capture
  -> refresh manager reports capture if stale
  -> fix the owning layer
  -> rerun the focused test

The Framework Layers

When working through the The Framework Layers stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours.

Layer                                           Purpose
AGENTS.md and .github/copilot-instructions.md   Tell agents how to behave
contexts/*.yml                                  Store command, strategy, graph, loop, and capture rules
contexts/capture-recipes.yml                    Defines repeatable capture workflows as YAML recipes
contexts/captures/                              Stores curated page context
specs/                                          Human-readable flow plans
fixtures/                                       Shared setup, auth, browser context, page instantiation
pages/                                          Locators and user actions
assertions/                                     Reusable web-first expectations
data/                                           Test data builders
tests/                                          Executable Playwright specs
graphs/                                         Generated source, coverage, duplicate, and visual reports

Source-Aware Graph Intelligence

When working through the Source-Aware Graph Intelligence stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours.

npm run source:manifest -- --root=path/to/app --output=source-manifest.json --url=http://localhost:3000 --name=my-app

scripts/source-adapters/generic-source.mjs

generic scanner creates baseline manifest
  -> agent reviews source, tests, captures, and graph gaps
  -> agent enriches missing auth, roles, states, and workflow intent
  -> capture workflow becomes curated reusable page context
  -> Playwright tests verify behavior
  -> graph reports remaining gaps and duplicate risks

npm run source:manifest:crm
npm run source:manifest:crm:check

npm run graph:view

graphs/index.html#coverage
graphs/index.html#candidates
graphs/index.html#inputs

Example: Manager Role Coverage Improved

When working through the Example Manager Role Coverage stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface. When working through the Example Manager Role Coverage stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

Before graph
  -> identify missing manager coverage
  -> generate/update manager capture workflow
  -> capture manager page states
  -> add page-object and assertion helpers
  -> add manager role tests
  -> rerun graph

Sample CRM test app:

The Sample CRM test app stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve.

How Teams Can Adopt This

The How Teams Can Adopt stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve.

Phase 1: Add Agent Instructions

The Phase 1 Add Agent stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve. The Phase 1 Add Agent stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

Phase 2: Align The Playwright Framework

For the Phase 2 Align The stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Authenticate at the gateway and re-authorize at the data plane. A bearer token alone is not a tenancy boundary.

specs/
tests/
fixtures/
pages/
assertions/
data/
contexts/captures/

Phase 3: Add Capture Workflows

For the Phase 3 Add Capture stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Authenticate at the gateway and re-authorize at the data plane. A bearer token alone is not a tenancy boundary.

npm install -g @playwright/cli@latest

npm run capture:recipes:generate
npm run capture:recipe -- --list
npm run capture:crm:manager -- --dry-run
npm run capture:crm:manager

Phase 4: Add Graph Review

For the Phase 4 Add Graph stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Authenticate at the gateway and re-authorize at the data plane. A bearer token alone is not a tenancy boundary. For the Phase 4 Add Graph stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

npm run graph:view

Phase 5: Use The Same Commands In CI

When working through the Phase 5 Use The stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours.

npm run typecheck
npm test
npm run test:smoke
npm run graph:view

Prompt Examples Teams Can Use

When working through the Prompt Examples Teams Can stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Cache stable system instructions and tool schemas. Re-sending identical preamble is a common source of burn.

Use the harness to add coverage for this role-based business flow.

check MCP access
refresh or read the source manifest
inspect existing specs, tests, captures, and graph coverage
use MCP agents or the relevant capture workflow
reuse framework layers
run focused Playwright tests
update the graph
report evidence

Use the harness to debug this failing test.

inspect Playwright output, trace, and artifacts
check existing captures for the affected page or state
refresh source manifest or affected capture only if stale
fix the owning layer
rerun focused verification

Use the harness to review missing coverage.

refresh or read the source manifest
run graph view
identify missing role, state, route, capture, and assertion coverage
check duplicate risks
recommend next flows
avoid creating tests unless asked

Use the harness to refactor repeated selectors and assertions.

inspect existing tests, page objects, and assertion helpers
preserve behavior
move repeated selectors to page objects
move repeated expectations to assertion helpers
run typecheck and focused tests
update graph if coverage ownership changed

What Teams Get

When working through the What Teams Get stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours. When working through the What Teams Get stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

Final Thought

The Final Thought stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve.

prompt
  -> mode router
  -> context
  -> capture workflow or MCP exploration
  -> framework layers
  -> Playwright verification
  -> evidence
  -> graph feedback

References

The References stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve.

Operational checklist

The Operational checklist stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope.

Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion.

Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve.

Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness.

Write a short runbook: how to rotate keys, how to drain the queue, how to roll back the last ingest.

Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

Before promoting the stack, freeze versions, capture a golden transcript for the critical path, and confirm rollback steps. Shared environments need rate limits, tenancy checks, and a clear owner for secret rotation. Prefer boring reliability over clever one-off demos.

Batch note for fa507de22687: keep provider keys out of the repo, set a per-session token ceiling, and store transcripts next to the eval fixtures so later model swaps stay comparable.