Beyond Dashboards: Building an Autonomous BigQuery Data Analyst Agent with

This walkthrough rebuilds the path from raw materials to a working system for: Beyond Dashboards: Building an Autonomous BigQuery Data Analyst Agent with Google ADK for Customer 360. The focus is operable steps, explicit checks, and code that you can drop into a repo without guessing intent. For the Overview stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.

1. The Last-Mile Bottleneck: Why Traditional BI and Naive Text-to-SQL Fail

When working through the 1 The Last-Mile Bottleneck stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.

Why Traditional BI Dashboards Break Down

When working through the Why Traditional BI Dashboards stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.

Why Naive “Text-to-SQL” LLM Wrappers Fail in the Enterprise

When working through the Why Naive Text-to-SQL LLM stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Cache stable system instructions and tool schemas. Re-sending identical preamble is a common source of burn. When working through the Why Naive Text-to-SQL LLM stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.

2. Agentic Architecture: The Google ADK Framework

The 2 Agentic Architecture The stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts.

3. Engineering Pillar 1: Knowledge Grounding & Vertex AI Context Caching

The 3 Engineering Pillar 1 stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts.

Compiling Markdown Dictionaries into the System Prompt

The Compiling Markdown Dictionaries into stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Budget tokens per turn and per session. Agentic tools expand context aggressively; hard caps keep demos from becoming surprise invoices. The Compiling Markdown Dictionaries into stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.

# c360_agent/config.py
import os
import glob

def compile_knowledge_base(knowledge_dir: str) -> str:
    """Recursively compiles markdown architecture guides into structured agent context."""
    context_chunks = []
    guide_files = glob.glob(os.path.join(knowledge_dir, "**/*.md"), recursive=True)

    for file_path in sorted(guide_files):
        with open(file_path, "r", encoding="utf-8") as f:
            rel_name = os.path.relpath(file_path, knowledge_dir)
            content = f.read()
            context_chunks.append(f"### KNOWLEDGE GUIDE: {rel_name}\n\n{content}\n")

    return "\n---\n".join(context_chunks)

KNOWLEDGE_CONTEXT = compile_knowledge_base("c360_agent/knowledge")

SYSTEM_PROMPT = f"""
You are an expert BigQuery Data Analyst Agent for an Enterprise Customer 360 (C360) Lakehouse.
Your mission is to answer business and analytical questions by generating and executing accurate, cost-effective SQL queries.

CRITICAL OPERATIONAL RULES:
1. TARGET CONFORMED MARTS FIRST: Always query `c360_curated_unified` for golden demographics or cross-system questions. Use Spoke marts (`c360_curated_*`) only for system-specific line-item grain.
2. MANDATORY DRY RUN: You MUST execute `dry_run_bq_query` on every generated SQL statement before calling `execute_bq_query`. Inspect syntax and bytes scanned.
3. CLUSTER AWARENESS: Always filter or join on clustered keys (`UnifiedCustomerDimId`, `UnifiedOrderDimId`) to minimize BigQuery slot consumption.
4. STRICT READ-ONLY: You are strictly forbidden from executing INSERT, UPDATE, DELETE, DROP, or ALTER statements.

DOMAIN ARCHITECTURE CONTEXT:
{KNOWLEDGE_CONTEXT}
"""

The Cost Game-Changer: Vertex AI Context Caching

For the The Cost Game-Changer Vertex stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness.

# c360_agent/agent.py
from google.adk.agents.llm_agent import Agent
from google.adk.apps import App
from google.adk.agents.context_cache_config import ContextCacheConfig
from .config import SYSTEM_PROMPT
from .tools import (
    execute_bq_query,
    dry_run_bq_query,
    get_table_schema,
    list_dataset_tables,
    get_table_preview
)

# 1. Initialize the Google ADK Agent
root_agent = Agent(
    name="c360_data_analyst",
    model="projects/enterprise-c360-prod/locations/us-central1/publishers/google/models/gemini-2.5-flash",
    description="Autonomous BigQuery Data Analyst for the Enterprise Customer 360 Lakehouse.",
    instruction=SYSTEM_PROMPT,
    tools=[
        dry_run_bq_query,
        execute_bq_query,
        get_table_schema,
        list_dataset_tables,
        get_table_preview
    ]
)

# 2. Wrap in App with Vertex AI Context Caching
app = App(
    name="c360_agent_app",
    root_agent=root_agent,
    context_cache_config=ContextCacheConfig(
        ttl_seconds=1800,   # Cache prompt for 30 minutes
        cache_intervals=10  # Reuse across up to 10 conversational turns
    )
)

4. Engineering Pillar 2: Safe, Cost-Guarded Tool Execution

For the 4 Engineering Pillar 2 stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Authenticate at the gateway and re-authorize at the data plane. A bearer token alone is not a tenancy boundary.

# c360_agent/tools.py
import subprocess
import re

def dry_run_bq_query(query: str) -> str:
    """Dry-runs a BigQuery SQL query to validate syntax and estimate bytes scanned."""
    clean_query = query.strip().upper()

    # Layer 1: Strict Read-Only Guard
    if not (clean_query.startswith("SELECT") or clean_query.startswith("WITH")):
        return "SECURITY ERROR: Only SELECT queries are permitted. DDL/DML is strictly forbidden."

    try:
        # Executes 'bq query --dry_run' to inspect execution planner without billing slots
        result = subprocess.run(
            ["bq", "query", "--use_legacy_sql=false", "--dry_run", query],
            capture_output=True,
            text=True,
            check=True
        )
        # bq CLI outputs estimated bytes scanned to stderr on dry run
        dry_run_output = result.stderr if result.stderr else result.stdout
        return f"DRY_RUN SUCCESS: {dry_run_output.strip()}"
    except subprocess.CalledProcessError as e:
        return f"DRY_RUN SYNTAX ERROR: {e.stderr.strip()}"


def execute_bq_query(query: str) -> str:
    """Executes a validated BigQuery SQL query and returns the formatted tabular results."""
    clean_query = query.strip().upper()

    if not (clean_query.startswith("SELECT") or clean_query.startswith("WITH")):
        return "SECURITY ERROR: Only SELECT queries are permitted."

    try:
        result = subprocess.run(
            ["bq", "query", "--use_legacy_sql=false", "--format=pretty", "--max_rows=50", query],
            capture_output=True,
            text=True,
            check=True
        )
        return result.stdout
    except subprocess.CalledProcessError as e:
        return f"EXECUTION ERROR: {e.stderr.strip()}"

5. Engineering Pillar 3: The Autonomous Self-Correction Loop

For the 5 Engineering Pillar 3 stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness. For the 5 Engineering Pillar 3 stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.

Real Production Trace: The Self-Healing Loop in Action

When working through the Real Production Trace The stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.

The Business User’s Question

When working through the The Business User s stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.

Step 1: Initial Attempt & Compiler Rejection

When working through the Step 1 Initial Attempt stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node. When working through the Step 1 Initial Attempt stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.

-- Attempt 1: Generated by Agent (Pre-Flight Dry Run)
SELECT
    SUM(c.total_spent) AS total_vip_revenue
FROM `enterprise-c360-prod.c360_curated_shopify.dim_customer` c
JOIN `enterprise-c360-prod.c360_curated_shopify.CustomerWarranties` w
    ON c.customer_id = w.customer_id;

Step 2: Autonomous Diagnosis & Self-Correction

The Step 2 Autonomous Diagnosis stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts.

  SELECT SUM(c.total_spent) AS total_vip_revenue
  FROM `enterprise-c360-prod.c360_curated_shopify.dim_customer` c
  JOIN `enterprise-c360-prod.c360_curated_shopify.CustomerWarranties` w
-     ON c.customer_id = w.customer_id;
+     ON c.UnifiedCustomerDimId = w.UnifiedCustomerDimId;

Step 3: Pre-Flight Cost Validation

The Step 3 Pre-Flight Cost stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts.

Step 4: Bounded Execution & Executive Synthesis

The Step 4 Bounded Execution stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts. The Step 4 Bounded Execution stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.

6. From Terminal CLI to Interactive Web UI

For the 6 From Terminal CLI stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness.

Mode 1: Terminal CLI for Data Engineers & CI/CD

For the Mode 1 Terminal CLI stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness.

python3 c360_agent/agent.py "Show me top 5 customers by SAP delivery volume in 2026"

Mode 2: Interactive Web Debug Console (adk web)

For the Mode 2 Interactive Web stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness. For the Mode 2 Interactive Web stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.

adk web --port 8001 c360_agent --reload

Summary: The Enterprise Autonomous Agent Checklist

When working through the Summary The Enterprise Autonomous stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.

The Trilogy Concluded: The Modern Enterprise Data Foundation

When working through the The Trilogy Concluded The stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.

Operational checklist

The Operational checklist stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope.

Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts.

Add a smoke test that exercises the critical path in CI with fixtures, not live paid APIs, whenever budgets allow.

Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.

Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts.