Agentic Analytics Is More Than Text-to-SQL: Lessons From Anthropic

Analytics slop is the new AI slop

A June 3, 2026 post from Anthropic on self-service analytics says in public what many data teams have been learning the hard way: agentic analytics is not the same as pointing Claude Code at a database and asking it questions.

Most people know the term AI slop. Analytics slop is its cousin. With a coding agent, anyone can now connect to a database and produce analytic reports. That's great for lowering the barrier to entry, but on the flip side, almost nobody checks the numbers. Dashboards arrive as .html files, Confluence pages or terminal screenshots, and few of them have been verified. The reason is a false sense of security. Since LLMs translate English into SQL so well, the answers must be right, or so the thinking goes.

That confidence is misplaced. When agentic analytics shrinks to query generation, the problem gets framed too narrowly and the surrounding infrastructure goes unnoticed, even though it determines whether a given answer helps or hurts. Anthropic's own design puts data foundations, sources of truth, skills and validation upstream of the moment Claude ever sees a query. The post lists classic data engineering practices as hard requirements, and it passes on lessons so other teams can skip the mistakes.

The post is an Enterprise AI piece, not a research paper, but it's one of the clearest production write-ups on what trustworthy agentic analytics takes. What follows works through its main ideas in order: why text-to-SQL is the wrong frame, where ambiguity comes from, what Anthropic built, what has to exist before a query is written, the role of the semantic layer and skills, why evals and maintenance matter, and where a team might begin.

Why text-to-SQL gets all the attention

Query generation is the part that can be demoed in thirty seconds. Ask a question in plain English, watch the model write a query, run it against the warehouse, and get back a number or a chart. It looks magical, especially to non-technical people who used to depend on an analytics team or pre-built dashboards.

A smooth explanation and a tidy number from Claude or Codex say nothing about whether the question was answered correctly. In Anthropic's words, aiming Claude at a warehouse can produce a false sense of precision, and that's the trap.

Take a stakeholder asking, "What was active customer revenue last month?" The analytic nuances start immediately:

• What makes a customer active?
• Which definition of customer is the official one?
• Is the figure gross or net revenue?
• Which date field applies?
• Which refunds, fraud filters or product exclusions should be applied?

Text-to-SQL only comes after those decisions are made. A reliable analytics agent therefore needs a whole infrastructure around it that steers its understanding of the business.

This echoes a well-known Google paper on hidden technical debt in machine learning systems, which showed that the model is a tiny box inside the much larger apparatus of production ML.

The figure above is the screenshot from that paper. The same picture carries over to analytics agents if the small black box is replaced with text-to-SQL.

Anthropic's answer lines up with that framing, in the line "Data is not software." Analytics work fails differently from software features. There's typically one correct answer from one correct source, yet no deterministic way to confirm it from the final output alone. And a system can write flawless SQL while missing the business meaning completely.

Anthropic also gives a set of examples contrasting the two paradigms, shown in the table below.

Anthropic sorts the resulting failures into three modes:

1. Concept or entity ambiguity. A user says "active users", "revenue" or "retention", and the agent must pick among several plausible implementations. The differences are often subtle enough that a wrong answer still looks professional.
2. Data staleness. Documentation, tables and business rules keep changing. A metric that held up three weeks ago can go quietly wrong once a schema shifts, a new exclusion rule lands or a dashboard is rebuilt.
3. Retrieval failure. Somewhere in the stack the correct answer exists, but there's too much to search. The model overlooks the right table, metric or document and confidently builds on something close but wrong.

The warehouse by itself can't resolve any of this. It stores data and not the complete meaning of the business, which is scattered over metric definitions, semantic models, caveats, team conventions, ownership and history. In the revenue example, writing a GROUP BY is easy. Deciding what "active" means is hard: one purchase within thirty days, paid non-fraud usage in a billing period, or a domain-specific threshold that lives only in the semantic layer.

So whether Claude can write SQL is hardly the interesting question, since it probably writes it better than most people. The useful question is what Anthropic had to build around Claude so these problems stopped dominating the outcome.

What Anthropic built

The answer is a blend of proven engineering practice and additional layers that agents make possible. In effect it's a full system with Claude as the interface on top and everything else living underneath. The post organizes it into four parts:

1. Data foundations, meaning canonical datasets, data models, transforms, tests, freshness checks and metadata.
2. Sources of truth, covering the semantic layer, lineage, business context and curated reference material that helps map a question onto governed meaning.
3. Skills, which are domain procedures telling the model how to work and not merely what data exists.
4. Validation, including offline evals, online checks, provenance and maintenance loops.

The screenshot above comes from Anthropic's post. Look at where SQL falls in the sequence: after all the hard decisions. When the model finally writes a query, the stack has already narrowed what the question means and which governed path deserves trust.

This structure accounts for the headline number better than the choice of model does. A quick read makes the 95 percent figure sound like a model benchmark, while a careful read shows a systems benchmark, given how much of the post describes the harness wrapped around Claude and how little describes Claude.

What has to exist before any SQL is written

Anthropic is explicit that ordinary data engineering still applies: dimensional modeling, freshness and completeness checks, consumption-ready canonical datasets, clear data dictionaries. None of it goes away because the interface became conversational. If anything the stakes rise, since the consumer is now an agent acting on a human's question. An analyst can sometimes sense that a table looks off, whereas an agent can scale a bad assumption far faster.

There's a bit of irony in a frontier company telling the world that good old data engineering is essential for agents. None of these practices are new, and any twenty-year-old book on data warehousing gives the same advice. They're still worth restating.

Canonical datasets

A small set of heavily governed logical models, each the single source of truth, reduces the number of plausible answers before retrieval even starts. Different aggregation levels may be needed, but the roll-ups should always read from that small base. The same holds for analytic agents: when they can reach only a few trusted datasets and know how those map to enriched dimensions, the chance of picking the wrong source drops sharply.

Metadata as a first-class product

Metadata hands the agent a working description of tables, columns, grain, owners and caveats. If nobody documents the model, the meaning of each column, and the lineage or join keys, the LLM papers over every gap with guesses.

Keeping models, docs, dashboards and skill files together

Colocation might simply mean a large monorepo, but Anthropic spells out something sharper. Almost all data code, from modeling and the semantic layer to reference docs and canonical dashboard definitions, sits in one repository guarded by CI checks on cross-layer integrity. A modeling change that would break a downstream dashboard or invalidate a documented metric gets flagged by CI, and the repair ships within the same pull request.

That undercuts the lazy story that AI analytics makes data engineering less important. The more questions route through models, the more visible the quality of the modeling layer becomes.

Other vendors show it as well. The dbt Semantic Layer exists so metric definitions are written once and reused by every tool. Microsoft advises preparing Power BI data for Copilot with AI-ready schemas, verified answers and explicit instructions, so the model replies from grounded context. Snowflake builds Cortex Analyst around semantic models and verified queries, with monitoring around the SQL generation step. Each product teaches the same thing: context prepared in advance is part of the answer.

The semantic layer is the agent's map

A semantic layer is an architectural bridge that turns raw, complex data into familiar business terms, metrics and relationships. It sits between data storage, such as Snowflake, Databricks or BigQuery, and consumption tools, such as Power BI, Tableau or AI agents. With one in place, you can answer what a column means, what type another has, how a metric is calculated, and how hierarchies and table relationships work.

The diagram above illustrates where that layer sits. Anthropic treats it as so central that its agents are structurally required to consult the semantic layer first. The operating principle is that the model starts from governed meaning and not from open-ended exploration of raw tables. Anthropic goes further and calls it bad practice to let agents invent the semantic layer themselves. Humans remain essential to keeping it good, and that comes from one of the world's frontier labs.

The industry is drifting toward the same view. Snowflake's Cortex Analyst uses a semantic model to connect business questions with database structure. Genie from Databricks has domain experts provide datasets, instructions, sample SQL and business context. dbt keeps metric logic in one place so downstream tools stop rebuilding it, and Microsoft moves verified answers and AI instructions into the BI model. Google's data science agent needed something similar, a pseudo-semantic layer named the Analyzer, before its results broke through.

Without that map the agent guesses. With it, the model spends its effort reasoning inside a constrained world and doesn't invent one from scratch.

Example queries add to the layer

Anthropic reports that a query corpus, meaning historical SQL from dashboards, notebooks and earlier analyses, can have a positive net effect. What works is not feeding it to the agent raw. Instead the corpus gets distilled into structured per-domain reference docs and reusable analysis patterns captured in skills. Query history is raw material for curation, not a source of truth the agent reads directly.

There's also a practical consequence for who owns what. Because the analytics agent leans on governed definitions, the people who maintain those definitions, usually analytics engineers and domain owners, become part of the agent's quality story. A disputed metric stops being an argument over a dashboard and becomes a defect in the shared layer, one that is fixed in a single place and picked up by every consumer, human or model. Teams that already work this way will find the move to agents less painful than teams whose metric logic is scattered across notebooks and BI tools.

Why skills move accuracy so much

A semantic layer explains what the world means but not how to act in it. It can't say what steps a good analyst takes when a stakeholder asks an ambiguous question that has two candidate sources and a known edge case in a single region.

In Anthropic's analytics system, skills work like encoded analyst playbooks and not like documentation. They hold procedural knowledge:

• the order in which sources get checked
• the point where the semantic layer should be used and where a fallback is allowed
• which clarifying questions to put to the stakeholder
• the caveats that apply in that domain
• a way to challenge the result before trusting it
• the provenance details that go into the final answer

Anthropic's strongest number comes from here. According to the skills section of the post, Claude never got above 21 percent on their analytics evals without skills. With the skill layer, accuracy rose past 95 percent in aggregate and hit roughly 99 percent in some domains.

That figure changes what the system should be called. If the largest jump comes from procedural guidance and not better SQL generation, then this isn't fundamentally a text-to-SQL product. It's an analytics workflow product with query execution inside it.

It also helps to remember what a skill is not. It isn't a longer system prompt, and it isn't a pile of example queries pasted in. It's a maintained artifact with a version, a trigger condition and a defined procedure, loaded when the question calls for it. That makes it reviewable in a pull request, testable against the eval set, and easy to retire when the underlying data model changes.

The image below shows the skill template Anthropic uses.

Taken together, these artifacts make up a full definition, and the appendix reproduces the whole template.

Evals and maintenance count as much as prompts

Even a well-grounded analytics agent decays over time. Once semantic layers, curated examples and skills exist, drift becomes the next problem: tables change, definitions move, dashboards are rebuilt and skills stop matching the source model. A procedure that was right yesterday turns into stale context tomorrow.

Anthropic gives a stark example: offline accuracy fell from about 95 percent to about 65 percent across a single month before skill maintenance began being treated as an engineering problem. That figure makes maintenance a reliability requirement instead of a nicety.

Analytics has a convenient property here, which is that evaluations can rest on ground-truth numbers any correct procedure should reproduce. Once that's defined, every other parameter of the system can be tuned against it. A team could try version two of a skill with Haiku, Sonnet or Opus at several thinking levels, compare accuracy, and pick whichever balances cost and quality best, whether that means enough accuracy at lower cost or the highest accuracy at any price.

A mature analytics system gets tested in a particular way: stored cases, explicit assertions, and a feedback loop that rewrites the playbook whenever the system gets something wrong. Databricks has a similar idea in its Genie documentation, which builds response inspection, benchmarks, trusted assets and curated instructions into the product.

Prompts matter less here than people tend to assume. Quality still counts, but with a weak semantic layer, stale examples, drifting skills and no evals, no heroic prompt will save the system for long. Agentic analytics works when the surrounding system is run like a maintained product, and not when a model gets raw access and is left to improvise.

Where a data team could start

The best place to begin is the simplest structure that could work, with complexity added gradually and only where results prove it pays off. Anthropic's guidance on building effective agents makes the same argument, and its analytics post follows it. Here are five steps:

1. Pick one domain and not the whole warehouse, such as revenue quality, support operations, growth funnels or finance reporting. Choose an area with a clear owner and recurring questions.
2. Establish a single governed source of truth. Prompts can wait until the domain has canonical datasets, written metric definitions, a named owner and basic freshness monitoring.
3. Put the semantic layer first and keep raw SQL as the fallback. When a question maps cleanly onto governed meaning, the agent shouldn't freestyle through raw tables.
4. Draft one thin skill or playbook that captures how a strong analyst works in that domain: clarify, pick sources, validate, report, and cite where the numbers came from.
5. Assemble a small eval set before going wide. Twenty to thirty real questions, with expected answers and known edge cases, teach more than a hundred internal demos.

One thing to avoid is wiring a frontier model straight to the warehouse and calling it a strategy. That approach teaches little beyond how convincing a wrong answer can sound.

Text-to-SQL keeps a place in the stack as one useful way to execute. The trouble is that it's too small a concept to describe the entire system. The subject is likely to divide teams into two groups: people who treat text-to-SQL as the product and people who see a single thin layer inside a wider analytics workflow. Questions worth asking: where do the biggest failures come from (ambiguity, retrieval, staleness or weak process), do the foundations an analytics agent needs already exist or would the project uncover hidden data-model debt, and what does the skill layer hold today, whether prompts, runbooks, SQL examples or analyst checklists.

Appendix: the full skill template

The complete skill template from Anthropic is below.

---
name: [warehouse-skill]
version: [x.y.z]
description: "IF the user asks to query [the company]'s data warehouse for any
  [list of business domains] question — THEN invoke this skill. DO NOT invoke
  for [adjacent engineering tasks] or questions with no data-warehouse component."
---

# [Warehouse] Skill Instructions

## Description
The single source of truth for safe and effective [warehouse] querying.
Referenced by other skills [listed] for query execution guidance.

Act as a Data Analyst, providing strategic insights and data-driven
recommendations but seek guidance along the way.

**Out-of-scope decisions**: [product areas, etc.] → surface data only,
state "decision is [owning team]'s call", do NOT take a position or author
code fixes.

## Executing queries
Priority:
1. **[Managed connection]** (if available): [query tool] / [schema tool]
2. **[CLI fallback]** (if installed): [default project, fallback project]
3. **Neither** — ask the user to authenticate, then stop

---

# Semantic Layer (REQUIRED first step)

The governed semantic layer is the **mandatory default path** for every data
question — same numbers as [the BI tool], joins/grain/filters baked in. Raw SQL
via the reference docs below is the **fallback**, used only after the
semantic-layer path is shown not to cover the ask.

## Required workflow
1. **Load** — [how to load the semantic layer in each runtime, with fallbacks]
2. **Discover** — search measures/dimensions by keyword; **always check
   segments** (the named canonical population filters — hand-rolled WHERE
   clauses for these are the dominant wrong-answer mode)
3. **Compile + run** — build the spec → compile to SQL → execute
4. **Fallback** — only if discovery finds no relevant metric or compile fails
   → raw SQL via `references/*.md` (PART 3 below)

> **Don't bail early.** Do NOT fall back to raw SQL on these grounds:
> - "[custom date filtering / cohorts]" → [covered by time-dimension specs]
> - "[needs a join]" → [the metric layer already encapsulates its joins]
> - [3–4 more pre-rebutted excuses agents use to skip the semantic layer]

### Date windows & timezone — decide before you query
- **As-of date vs trailing-N days**: [convention for each]
- **"Last week/month"** → the last *complete* calendar week/month, not trailing-7/30
- **Timezone default**: [TZ]; [exception for certain reporting rollups]
- **Freshness lag**: [some] tables settle late — anchor on MAX(date), not "yesterday"

---

# PART 1: MUST KNOW (Read First for Every Request)

## 🚀 Quick Start Workflow
1. **Check for red flags first**: [restricted/PII requests, gated domains,
   high-stakes asks that need extra validation]
2. **Out of scope — escalate, don't guess**: [access requests, pipeline
   troubleshooting, stale dashboards, root-cause assertions, product/pricing
   recommendations] → redirect to [the owning team], don't answer
3. **Clarify the request**: time period, segment, the business decision it informs
4. **Check for existing dashboards**: [per-domain dashboard catalogs]
5. **Identify the data source**: [navigation map below; prefer governed/aggregated tables]
6. **Execute the analysis**: [required filters + adversarial review]
7. **Deliver insights**: show methodology, differentiate observations from interpretations

## 🏢 Business Context

### Entity Disambiguation (MUST CLARIFY)
- **"[Term A]" can mean**: [entity 1] or [entity 2] — always clarify which
- **"[Term B]" can mean**: [entity 1] → [entity 2] → [entity 3] (one-to-many chain)
- **"Users"**: [which identifier gives accurate counts, and which ones inflate them]

### Business Terminology
- [Current product names vs deprecated aliases that still appear as frozen
  values in the data layer — write with the new names, filter with the old]
- [Key internal acronyms]
- **[Headline metric] calculations**: [monthly / default window / leading indicator]
- **Unfamiliar terms — search [internal docs], don't guess**

### Data Integrity Requirements ⚠️
- **NEVER**: make up data/columns; make speculative assertions beyond what data shows
- **ALWAYS**: use safe division; differentiate observations ("data shows X")
  from interpretations ("this suggests Y"); flag limitations

---

# PART 2: HOW TO DO (Follow During Execution)

## 🔧 Technical Execution Guide
- [Managed-connection tools and CLI invocation details]
- **PII protection**: for restricted data, return the SQL for the user to run
  themselves — do not return results

## 📊 Analysis Best Practices Guide
1. Clarify the ask before querying
2. Show your work (filters, inclusions/exclusions, freshness)
3. Clarify denominators
4. Consider sample bias
5. Connect to business impact
6. **Adversarial SQL review (MANDATORY)** — spawn the [sql-reviewer] sub-agent
   for every query before the final answer; blocking findings must be fixed
   and re-reviewed; do not self-certify
7. **Report with provenance** — every answer ends with a footer:
   > **Source:** [semantic layer | governed table | raw exploration] ·
   > **Confidence:** [tier] · **Reviewed:** [reviewer ✓, round N] ·
   > **Freshness:** [max date in the data] · **Owner:** [owning team]

---

# PART 3: DATA REFERENCES & RESOURCES

## 📚 Knowledge Base Navigation
### [Domain A] → `references/[domain_a].md`
- **Use for**: [kinds of questions]
- **Key tables**: [...]
- **Dashboards**: `references/[domain_a]_dashboards.json`

### [Domain B] → `references/[domain_b].md`
- **Use for**: [...]

[... one entry per business domain — a few dozen in total ...]

## ⚠️ Troubleshooting Guide

### When Information Is Missing
- [missing tables / access denied / outdated docs / unknown enum values → what to do]

### Field Naming Gotchas
- Use `[field_x_v2]` NOT `[field_x]`
- [Two similarly-named tables report the same metric at different grains — which to use]
- [Which of two plausible sources is canonical for the headline metric]
- [… a dozen more hard-won one-liners …]