From RAG Agent Dreams to a VBA Prototype: Separating Raw and Derived Data

The plan was modest: build a small version of Merge, a personal AI agent. Experiments with RAG, LangChain, LangGraph, embeddings and vector stores were already underway, and the original idea was to give the agent long-term memory and access to internal documents. Then a different problem got in the way. Freelance project management had turned chaotic, which suggested writing a small VBA tool first. That looked easy and wasn't. Along the way the VBA prototype turned into something else, a way to study the business logic that an AI agent might one day inherit.

Projects that refuse a clean lifecycle

Freelance work doesn't follow a tidy path. A landing page ships, then the client wants a redesign. The business direction shifts, payment integration appears, then LINE automation, then a related project. Financially, one task can already be finished and invoiced while the wider engagement is still running. Maintenance contracts are odder still, because many simply continue until the client says they no longer need them.

One status field can't describe that. Several independent state dimensions can:

billing_status = closed
task_status = active
invoice_status = pending

Read together, this says the amount is finalized, the task is still active and the invoice hasn't gone out. Twenty hand-defined combined states would have been needed otherwise. Independent fields keep the model small.

Separating responsibilities

After a few days of thought, a layering rule emerged:

Raw Data
   ↓
Small Logic
   ↓
Secondary Data
   ↓
END

Primary data holds facts. Secondary data holds calculated or transformed information meant for human eyes. One firm rule goes with it: no tertiary data. A secondary sheet must never feed another calculation. When a new result is needed, the work goes back to raw data and produces another secondary set. That could mean many secondary sheets eventually, and that is acceptable. Twenty independent views beat a chain like this:

Raw
 ↓
Secondary A
 ↓
Secondary B
 ↓
Secondary C
 ↓
Dashboard

because a single bug in the middle spreads everywhere. The working principle is that more sheets are fine, but deeper dependencies are not.

A strict workbook layout

Sheets are split by responsibility, using a prefix convention:

D_ = Raw Data
M_ = Master Data
S_ = System Rules
V_ = Secondary Views

The raw data sheets look like this:

D_Task
D_Point
D_Pharmacy

and the master sheets like this:

M_Company
M_Project
M_Contact
M_Product
M_GroupS_SystemV_TaskList
V_MonthlyBilling
V_CurrentSummary

The D_ sheets are the source of truth. The M_ sheets are business dictionaries covering companies, products, projects and contacts. S_System stores the VBA-specific configuration: the tax rate, what each status means, the hourly rate and income goals. Sheets with the V_ prefix are throwaway output. Any of them can be rebuilt on demand, and they exist for people to read, not for other formulas to rely on.

No direct edits

A second rule followed: normal operation should never need anyone to edit raw data or master sheets by hand. UserForms handle creating, updating and logically deleting records. Deletion isn't physical. A record is flagged instead:

record_status = deleted

The row stays in the system, which should make history easier to trace and cut accidental data loss.

VBA as an observation tool

This was the point where the whole project changed shape. VBA was first meant to automate the workflow. Its better role may be to observe it. The system can compute expected project income and expected pharmacist income, how much extra income is still needed, what remains on each project, which invoices are still unsent, where each project stands and how many points have been spent.

Looking at those numbers leads to decisions, such as concluding that extra pharmacist shifts aren't needed next month. Ask Merge how that decision was made and the honest answer today is something like "experience". That is exactly why the decision isn't automated yet. The system should show the same information consistently, and after enough real use, patterns in the decisions may become visible. Those patterns can then move into Python and finally into the agent. The possible progression:

Raw Data
   ↓
Python Logic
   ↓
Secondary Information
   ↓
AI Agent
   ↕
Human adjustment through Slack

So the VBA version isn't necessarily the final application. It works as a laboratory for finding out what the final application needs to know.

Different tools, different jobs

Forcing everything into one system would be a mistake. Excel covers numbers, billing, statuses, records and financial views. Notion holds detailed tasks, notes and project information. A physical whiteboard carries things to remember or notice.

The longer-term picture combines two views, one laid out like a report and one like a timeline, with Merge living in Slack as the main place to talk. A physical Merge device could later read important Slack messages aloud with text-to-speech. At first it would only produce output, with replies typed into Slack, since speech recognition is awkward outdoors and earlier experiments led to talking unnaturally loudly at a computer. Direct voice conversation with the physical device is where a Raspberry Pi would come back in.

The UI turns into a blackboard

After days of architecture talk, work began on the first UserForm, which manages groups, companies, projects, contacts and products. The initial attitude was that it was only for personal use and looks didn't matter. Then a funny font got picked, the background changed, white separator lines appeared, and the serious master-data form became a blackboard-style interface. Merge's verdict was that it smelled like MergeLab, and the honest reaction was that it was kind of likeable.

Internally everything is strict: dependencies, raw data ownership, logical deletion, and the ban on tertiary data. Visually it looks like someone teaching database normalization on a blackboard at midnight. The VBA still doesn't work, but the blackboard is at least consistent.

Lessons

The biggest lesson is that automation shouldn't start with automation. First comes sorting out which information is factual, which is derived and which decisions are still human judgment.

Fewer sheets don't automatically mean less complexity. Many independent secondary views can be easier to understand than a compact setup with deep chains of derived data.

States of different kinds belong in different fields. Instead of many combined statuses, independent fields can describe far more complex real situations:

task_status
billing_status
invoice_status
record_status

Business logic should stay out of the UI. A UserForm collects input and calls reusable logic, the worksheet triggers refreshes and shows results, and calculations live in small reusable modules. That makes a later migration of the prototype easier.

Most of all, the prototype can expose hidden decision rules. It doesn't have to make every decision today. It only has to make those decisions observable.

Next steps

The immediate job is to make the VBA do something, beginning with MasterManagementForm, which registers and manages groups, companies, projects, contacts and products. Its basic flow:

Input
 ↓
Validation
 ↓
Generate ID
 ↓
Create / Update / Logical Delete
 ↓
Master Data

Once master data works, separate input forms follow for the raw sheets:

D_Task
D_Point
D_Pharmacy

After that, the first secondary views can be generated. The most important ones for now:

V_TaskList
V_MonthlyBilling
V_CurrentSummary

When those are usable in real work, it will be clear which views actually get opened and which were only imagined. Only once that workflow is stable does a proper rebuild with Python, Slack integration and an AI agent become a serious option.

So the sequence so far: an attempt to build a RAG-powered agent, then Excel, then several days of drawing data boundaries, then a blackboard. Software development is going well.