Building a Modular JavaScript Automation Toolkit from Scratch

1. A Recurring Problem Became the Starting Point

At some point you notice you're doing the same kind of task again and again.

Load up a page.

Scan it for something useful.

Pull out the relevant bits.

Store them somewhere for later.

Move on to the next page.

Start the cycle again.

None of these steps is hard on its own, but the sheer repetition of them is absurd.

So rather than picking up yet another JavaScript framework just to build another dashboard, a more useful path is building something you can actually put to work: a JavaScript automation and data-collection tool.

The idea behind it is straightforward:

Hand the program a task → let JavaScript take care of the repetitive part → get back structured results.

An early version of such a tool only needs a handful of technologies:

• Node.js
• Playwright
• Cheerio
• SQLite
• Commander

That combination is enough to turn a small script into something that starts resembling an actual product.

2. Playwright as the First Building Block

The goal was to let JavaScript drive a real browser.

Playwright makes that far easier than expected.

const { chromium } = require("playwright");

async function visitWebsite(url) {
    const browser = await chromium.launch({
        headless: false
    });

    const page = await browser.newPage();

    await page.goto(url, {
        waitUntil: "domcontentloaded"
    });

    console.log(
        "Page title:",
        await page.title()
    );

    console.log(
        "Current URL:",
        page.url()
    );

    await browser.close();
}

visitWebsite(
    "https://example.com"
);

The first time you run something like this, it feels almost too easy.

JavaScript opens a browser.

JavaScript navigates to a page.

JavaScript reads what's on it.

JavaScript shuts the browser down.

That alone gives you a base you can build on for browser testing, monitoring, repetitive workflows, and general automation.

3. Moving From Coordinates to Elements

One thing worth avoiding from the start is fragile automation.

Telling your program something like:

Click at this exact location.

means the automation can break the moment the layout shifts even slightly.

A better approach is writing code that describes what a user actually interacts with, rather than where things happen to sit on screen.

async function searchPage(page, query) {
    await page
        .getByRole("textbox")
        .fill(query);

    await page
        .getByRole("button", {
            name: "Search"
        })
        .click();

    await page.waitForLoadState(
        "domcontentloaded"
    );
}

This is far easier to maintain over time.

The code isn't saying:

Click whatever happens to be at coordinate 742, 381.

It's saying:

Find the textbox and the search button.

This is a small design choice, but it makes browser automation dramatically less painful down the line.

4. Extracting Data From the Page

Once the browser can move around a page on its own, the next step is having it gather information.

Picture, for instance, a page filled with product cards.

async function extractProducts(page) {
    return page
        .locator(".product-card")
        .evaluateAll(cards => {
            return cards.map(card => {

                const name =
                    card
                        .querySelector(".product-name")
                        ?.textContent
                        ?.trim();

                const price =
                    card
                        .querySelector(".price")
                        ?.textContent
                        ?.trim();

                return {
                    name,
                    price
                };
            });
        });
}

At this point the browser isn't just visiting pages anymore.

It's converting what's on the page into JavaScript objects.

That makes the data usable for further processing.

const products =
    await extractProducts(page);

console.log(
    JSON.stringify(
        products,
        null,
        2
    )
);

Once the information has real structure, you can store it, compare it, run analysis on it, or pass it into another part of the application.

5. Bringing In Cheerio for HTML Parsing

Playwright shines when you genuinely need a browser running.

But often you already have the HTML in hand and don't need to spin up Chromium at all.

That's where Cheerio earns its place.

const cheerio = require("cheerio");

function parseProducts(html) {
    const $ = cheerio.load(html);

    const products = [];

    $(".product-card").each(
        (_, element) => {

            const name = $(element)
                .find(".product-name")
                .text()
                .trim();

            const price = $(element)
                .find(".price")
                .text()
                .trim();

            products.push({
                name,
                price
            });
        }
    );

    return products;
}

Having both tools on hand is a real advantage.

Playwright handles browser interaction.

Cheerio handles lightweight HTML parsing.

That way, a full browser instance is only used when it's actually necessary, and a simple parser covers the rest.

6. Giving the Automation a Memory With SQLite

Storage was the next challenge to solve.

If a script gathers data today, where does that data live tomorrow?

The answer was to bring in SQLite.

const sqlite3 = require("sqlite3").verbose();

const db = new sqlite3.Database(
    "automation.db"
);

db.run(`
    CREATE TABLE IF NOT EXISTS products (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        name TEXT NOT NULL,
        price TEXT,
        source TEXT,
        created_at DATETIME
            DEFAULT CURRENT_TIMESTAMP
    )
`);

From there, a dedicated function handled saving each result.

function saveProduct(product, source) {
    return new Promise(
        (resolve, reject) => {

            db.run(
                `
                INSERT INTO products
                (name, price, source)
                VALUES (?, ?, ?)
                `,
                [
                    product.name,
                    product.price,
                    source
                ],
                error => {

                    if (error) {
                        reject(error);
                        return;
                    }

                    resolve();
                }
            );
        }
    );
}

This addition changed the shape of the project in a meaningful way.

With a database in place, historical records could accumulate over time. That opened the door to answering practical questions like:

• What changed?
• What showed up recently?
• What disappeared?
• When was this data last collected?
• Which source produced a given record?

Automation becomes far more valuable once it can remember what happened before.

7. Assembling a Reusable Workflow Engine

By this stage, the project had several independent pieces: browser control, HTML parsing, and database persistence. What it lacked was structure — there was a real risk of ending up with loose functions scattered everywhere instead of a coherent system.

To fix that, a workflow class brought everything together.

class AutomationWorkflow {

    constructor(browser) {
        this.browser = browser;
        this.page = null;
    }

    async start() {
        this.page =
            await this.browser.newPage();
    }

    async visit(url) {
        await this.page.goto(url, {
            waitUntil: "domcontentloaded"
        });
    }

    async search(query) {
        await this.page
            .getByRole("textbox")
            .fill(query);

        await this.page
            .getByRole("button", {
                name: "Search"
            })
            .click();
    }

    async getTitle() {
        return this.page.title();
    }

    async close() {
        await this.page.close();
    }
}

With that class in place, a complete task becomes straightforward to follow from start to finish.

const browser =
    await chromium.launch({
        headless: false
    });

const workflow =
    new AutomationWorkflow(
        browser
    );

await workflow.start();

await workflow.visit(
    "https://example.com"
);

await workflow.search(
    "JavaScript automation"
);

console.log(
    await workflow.getTitle()
);

await workflow.close();

await browser.close();

This is the point where object-oriented design started paying off. The class instance models the workflow itself, and its methods represent the individual operations. The rest of the codebase doesn't need to know how each step is implemented internally.

8. Handling Failures With Retry Logic

Automation scripts have a habit of running flawlessly nine times and then failing on the tenth. The network slows down, a page isn't fully loaded, a server hiccups, or an element takes longer than expected to render.

To deal with this, a generic retry helper was introduced.

async function retry(
    operation,
    attempts = 3,
    delay = 2000
) {
    let lastError;

    for (
        let attempt = 1;
        attempt <= attempts;
        attempt++
    ) {
        try {
            return await operation();

        } catch (error) {

            lastError = error;

            console.log(
                `Attempt ${attempt} failed.`
            );

            if (
                attempt < attempts
            ) {
                await new Promise(
                    resolve =>
                        setTimeout(
                            resolve,
                            delay
                        )
                );
            }
        }
    }

    throw lastError;
}

That utility made it possible to wrap critical steps with automatic retries.

await retry(
    async () => {
        await page.goto(
            "https://example.com",
            {
                waitUntil:
                    "domcontentloaded"
            }
        );
    },
    3,
    1500
);

The takeaway was straightforward: production-grade automation has to plan for failure as a normal event, not an exception. Tutorials tend to assume a perfectly cooperative network. Real systems can't afford to make that assumption.

9. Wrapping It in a Command-Line Interface

At some point, opening the source file every time just to swap out a URL became tiresome.

The goal shifted toward making the tool behave like a proper command-line utility.

Commander made that easy to achieve.

const { Command } = require("commander");

const program = new Command();

program
    .name("webpilot")
    .description(
        "JavaScript automation toolkit"
    )
    .version("1.0.0");

program
    .command("visit")
    .description(
        "Open a webpage"
    )
    .argument("<url>")
    .action(async url => {

        const browser =
            await chromium.launch({
                headless: false
            });

        const page =
            await browser.newPage();

        await page.goto(url);

        console.log(
            await page.title()
        );

        await browser.close();
    });

program.parseAsync();

With that in place, the tool could be launched directly from a terminal.

node webpilot.js visit https://example.com

It looks like a minor adjustment, but it fundamentally changes how you interact with the software.

Instead of editing the program every time you need something,

you simply run it.

10. Treating AI as the Front End

That raised a new question:

Why should the user need to memorize the exact command syntax at all?

Rather than typing something like:

node webpilot.js screenshot https://example.com

a person could just describe the intent in plain language:

"Grab a screenshot of the homepage."

An AI layer could translate that sentence into a structured task object.

const task = {
    action: "screenshot",
    url: "https://example.com",
    output: "homepage.png"
};

However, the AI would never be given permission to run arbitrary JavaScript directly.

Every requested action would first pass through a validation step.

const allowedActions = new Set([
    "visit",
    "search",
    "screenshot",
    "download"
]);

function validateTask(task) {

    if (
        !allowedActions.has(
            task.action
        )
    ) {
        throw new Error(
            "Unsupported action."
        );
    }

    if (
        task.url &&
        !task.url.startsWith("https://")
    ) {
        throw new Error(
            "Invalid URL."
        );
    }

    return true;
}

This produces a clean separation of responsibilities:

The AI interprets what the user wants.

The JavaScript layer decides what is actually allowed.

Playwright carries out only the approved action.

That kind of layered design is a much safer approach than handing an AI model direct, unrestricted control over the machine.

11. Reframing the Project as a Sellable Product

At this stage, the focus moved away from the underlying libraries and toward the actual customer.

The pitch was never going to be:

"A browser automation app built with Playwright and JavaScript."

No one goes looking for Playwright specifically — they're looking for an outcome.

So the offering became the outcome itself. A few examples:

Website Monitoring

Businesses can track their own websites and get notified when key pages change or go down.

Automated QA

Engineering teams can run consistent, repeatable browser checks against their own applications.

Internal Workflow Automation

Organizations can automate repetitive browser-based tasks within tools they already have access to.

Reporting Automation

A scheduled job can gather approved data, save it, and compile it into a report automatically.

Agency Automation

An agency can design custom automation pipelines for clients and bill for setup and ongoing support.

Pricing could take several forms:

• A flat one-time setup charge
• Recurring monthly maintenance fees
• Pricing per individual workflow
• Team-based licensing
• Custom integration work
• Hosted, subscription-based access

The key is anchoring the offering to a concrete, specific problem rather than the technology stack.

12. Small Scripts Can Grow Into Real Products

By the end, the overall system architecture looked roughly like this:

User
                   │
                   ▼
             CLI / AI Input
                   │
                   ▼
             Task Validator
                   │
                   ▼
             Workflow Engine
                   │
          ┌────────┼────────┐
          ▼        ▼        ▼
     Playwright  Cheerio  SQLite
          │        │        │
          └────────┼────────┘
                   ▼
              Result Data
                   │
                   ▼
             Report / API

That's the part worth sitting with.

The underlying issue was never really about browser automation as such.

It was about eliminating repetitive manual work.

The libraries — Playwright, Cheerio, SQLite, Commander — were just the mechanisms for converting that repetition into working software.

That mindset now shapes how new JavaScript projects get approached.

Whenever a sequence of manual steps starts repeating for the twentieth time, the instinct isn't:

"Time to reach for a new framework."

It's:

"Could this workflow be turned into a function?"

If the answer is yes, that's the seed of an automation project.

And when that automation saves someone enough time and effort, it can grow into something worth selling.