DS-STAR vs General Coding Agents: Google's Data Analysis Harness Explained

General-purpose coding agents are impressive at everyday engineering, yet they tend to stall when pushed toward serious data science. A typical analysis stops at a few averages. They're useful, but there are no distribution comparisons, no p-values or confidence intervals, and little attention to data quality. It's like asking a generalist to do a specialist's job.

A Google paper, DS-STAR, tackles exactly that gap. It argues that the gains come from the framework and system design around a model, not from a bigger frontier model. The headline figures make the point. Gemini 2.5 Pro on its own scores 12.70% on the hard-level DABStep tasks. Wrapped in DS-STAR, the same model scores 45.24%, a jump of about 32 percentage points. The model didn't change. The system around it did.

The same reasoning explains why some coding tools beat others without always holding the best underlying model: how the system is wired matters as much as the weights. DS-STAR applies that thinking to data science, and what follows takes the system apart piece by piece.

The paper behind DS-STAR

The paper, simply called DS-STAR (a data science agent for diverse tasks), is by Jaehyun Nam (KAIST) with Jinsung Yoon, Jiefeng Chen, Raj Sinha and Tomas Pfister of Google Cloud. It first appeared on arXiv in September 2025 and reached its fourth version in February 2026. Google also published a shorter write-up on its research blog.

By the authors' description, DS-STAR is a data science agent designed to process and integrate data across diverse, heterogeneous formats, and to go beyond simple question answering by producing comprehensive research reports for open-ended queries.

Here is the plan for what follows:

• The problem the paper targets, and why single-pass approaches fail on real data science work.
• How the system is layered: DS-STAR handles precisely stated questions, DS-STAR+ takes on open-ended research.
• A close look at DS-STAR's seven modules, their formulas and the full iterative algorithm.
• DS-STAR+ and how decomposition, a writer and a refinement loop build structured reports.
• The prompts Google published in its appendix, and what the design choices reveal.
• Ablation tests, to see which component accounts for most of the 32-point gain.
• How the number of iterations grows with task difficulty.
• A sample report DS-STAR+ produced on a payments dataset.
• Limitations, including the lack of an official release.

What DS-STAR sets out to do for data analysis

The stated ambition is large: a system able to answer any data science question across heterogeneous formats and open-ended queries.

The data is what makes that hard. Real work rarely means one tidy CSV and a precisely worded question. It means several structured datasets that need multi-step reasoning, domain knowledge buried in rules, and questions that no single pass of code generation can settle. The DABStep benchmark, from Adyen and Hugging Face, captures this. It has 450 tasks, of which 72 are easy (one dataset, little context needed) and 378 are hard (several datasets, domain knowledge required, no one-shot solution). KramaBench goes further, with 104 challenges across 1,700 files and 24 data sources in 6 domains, where the agent must find out for itself which files matter.

Both benchmarks are built to defeat the "write one block of code and submit the answer" approach, which is precisely what DS-STAR replaces.

Even a narrower setting, such as one internal database and a well-defined question, benefits from these architecture choices. A system that copes with heterogeneous chaos will manage clean data. The reverse doesn't hold.

How DS-STAR is put together

There are really two systems, one built on top of the other. DS-STAR is the inner engine of DS-STAR+. The outer layer, DS-STAR+, splits an open-ended query into focused sub-questions and hands each to DS-STAR. A writer agent then synthesises the answers into a report, and an evaluator can trigger further rounds of analysis.

DS-STAR in depth

The paper's own diagram is decent, but it leaves out a few components that help in following the order of the algorithm, so it's worth reading the pipeline module by module in the order data flows through it.

The seven modules

DS-STAR consists of seven named agents.

Module 1, the Analyzer. Before any planning, DS-STAR profiles every data file it receives. For each file it generates a Python script, executes it and stores the printed output as a structured description, rather like a small indexer or a lightweight RAG layer. One notable detail is that the description comes from deterministic Python code, not from an LLM deciding freely how to inspect a file. The paper's appendix includes a sample script that analyses a JSON file together with the output it produces. Google also publishes the prompt for each module. This is the Analyzer's:

You are an expert data analysist.
Generate a Python code that loads and describes the content of {filename}.

# Requirement
- The file can both unstructured or structured data.
- If there are too many structured data, print out just few examples.
- Print out essential informations. For example, print out all the column names.
- The Python code should print out the content of {filename}.
- The code should be a single-file Python program that is self-contained and can be
executed as-is.
- Your response should only contain a single code block.
- Important: You should not include dummy contents since we will debug if error occurs.
- Do not use try: and except: to prevent error. I will debug it later.

Module 2, the Planner. Given the data descriptions and the user's question, the Planner proposes only the first step of an execution plan, not the whole thing at once. It runs in two modes, planner_init for the first step with no earlier plan, and planner_next for later steps, which sees the current plan and earlier execution results. This is the prompt that starts a plan:

You are an expert data analysist.
In order to answer factoid questions based on the given data, you have to first plan
effectively.

# Question
{question}

# Given data: {filenames}
{filenames #1}
{summaries #1}
...
{filenames #N}
{summaries #N}

# Your task
- Suggest your very first step to answer the question above.
- Your first step does not need to be sufficient to answer the question.
- Just propose a very simple initial step, which can act as a good starting point to
answer the question.
- Your response should only contain an initial step.

For subsequent steps the prompt changes. It now includes the previous steps and what they produced, which gives the system a memory of what has been attempted and supplies deeper context:

You are an expert data analysist.
In order to answer factoid questions based on the given data, you have to first plan
effectively.

Your task is to suggest next plan to do to answer the question.

# Question
{question}

# Given data: {filenames}
{filenames #1}
{summaries #1}
...
{filenames #N}
{summaries #N}

# Current plans
1. {Step 1}
...
k. {Step k}

# Obtained results from the current plans:
{result}

# Your task
- Suggest your next step to answer the question above.
- Your next step does not need to be sufficient to answer the question, but if it
requires only final simple last step you may suggest it.
- Just propose a very simple next step, which can act as a good intermediate point to
answer the question.
- Of course your response can be a plan which could directly answer the question.
- Your response should only contain an next step without any explanation.

Module 3, the Coder. It takes the current plan step and writes executable Python for it. Like the Planner it has two modes: coder_init for the first block and coder_next for later ones, aware of earlier results. Output is a single self-contained script. The initial prompt:

# Given data:
{filenames}
{filenames #1}
{summaries #1}
...
{filenames #N}
{summaries #N}

# Plan
{plan}

# Your task
- Implement the plan with the given data.
- Your response should be a single markdown Python code (wrapped in ```).
- There should be no additional headings or text in your response.

And the prompt for coding each subsequent action:

You are an expert data analysist.
Your task is to implement the next plan with the given data.

# Given data:
{filenames}
{filenames #1}
{summaries #1}
...
{filenames #N}
{summaries #N}

# Base code
```python
{base_code}
```

# Previous plans
1. {Step 1}
...
k. {Step k}

# Current plan to implement
{Step k+1}

# Your task
- Implement the current plan with the given data.
- The implementation should be done based on the base code.
- The base code is an implementation of the previous plans.
- Your response should be a single markdown Python code (wrapped in ```).
- There should be no additional headings or text in your response.

Module 4, the Debugger. The Coder's output runs against the real data files. If execution fails, whether from a syntax error, a missing column or a type mismatch, the Debugger takes over. It receives the broken code and the error message and produces a corrected script. The paper's Figure 1 doesn't give it much prominence, yet it appears in the algorithm, sitting between the Coder and the Verifier and dealing with execution failures before they ever reach plan evaluation. It uses two prompts. The first condenses the error report, since tracebacks can be very long:

# --> PROMPT TO SUMMARISE THE ERROR
# Error report
{bug}

# Your task
- Remove all unnecessary parts of the above error report.
- We are now running {filename}.py. Do not remove where the error occurred.

The second does the actual repair:

# --> PROMPT TO FIX THE ERROR
# Code with an error:
```python
{code}
```

# Error:
{bug}

# Your task
- Please revise the code to fix the error.
- Provide the improved, self-contained Python script again.
- There should be no additional headings or text in your response.
- Do not include dummy contents since we will debug if error occurs.
- All files/documents are in `data/` directory.

Module 5, the Verifier. Once code runs successfully, the Verifier judges whether the plan so far is enough to answer the original question. Its verdict is binary: sufficient or insufficient. The judgment draws on the cumulative plan, the user's query, the solution code implementing that plan and its execution result, using this prompt:

You are an expert data analysist.
Your task is to check whether the current plan and its code implementation is enough to
answer the question.

# Plan
1. {Step 1}
...
k. {Step k}

# Code
```python
{code}
```

# Execution result of code
{result}

# Question
{question}

# Your task
- Verify whether the current plan and its code implementation is enough to answer the
question.
- Your response should be one of 'Yes' or 'No'.
- If it is enough to answer the question, please answer 'Yes'.
- Otherwise, please answer 'No'.

Module 6, the Router. It acts only when the Verifier says insufficient. The Router reviews the plan, the question and the execution results, then chooses between two options. Either a new step is appended at the end ("Add Step"), or a particular existing step needs correcting ("Replace Step K", with K as the step index). The first fits the case where the question simply wasn't covered yet. The second matters because the system advances one step at a time: it can fix a single faulty step without rewriting the entire plan.

You are an expert data analysist.
Since current plan is insufficient to answer the question, your task is to decide how to
refine the plan to answer the question.

# Question
{question}

# Given data:
{filenames}
{filenames #1}
{summaries #1}
...
{filenames #N}
{summaries #N}

# Current plans
1. {Step 1}
...
k. {Step k}

# Obtained results from the current plans:
{result}

# Your task
- If you think one of the steps of current plans is wrong, answer among the following
options: Step 1, Step 2, ..., Step K.
- If you think we should perform new NEXT step, answer as 'Add Step'.
- Your response should only be Step 1 - Step K or Add Step.

Module 7, the Finalizer. When the Verifier returns sufficient, the Finalizer pulls the relevant result out of the accumulated execution history and formats it as the answer to the original question.

You are an expert data analysist.
You will answer factoid question by loading and referencing the files/documents listed
below. You also have a reference code.

Your task is to make solution code to print out the answer of the question following the
given guideline.

# Given data: {filenames}
{filenames #1}
{summaries #1}
...
{filenames #N}
{summaries #N}

# Reference code
```python{code}
```
# Execution result of reference code
{result}

# Question
{question}

# Guidelines
{guidelines}

# Your task
- Modify the solution code to print out answer to follow the give guidelines.
- If the answer can be obtained from the execution result of the reference code, just
generate a Python code that prints out the desired answer.
- The code should be a single-file Python program that is self-contained and can be
executed as-is.
- Your response should only contain a single code block.
- Do not include dummy contents since we will debug if error occurs.
- Do not use try: and except: to prevent error. I will debug it later.
- All files/documents are in `data/` directory.

One point to keep in mind: the pipeline is sequential, not parallel. Each agent's output feeds the next, so altering any one of them changes what its successor sees.

The formulas

An interesting feature of the paper is that it expresses the algorithm in formulas. The message isn't "here is how our prompts fit together because we worked it out", but rather "this system consists of logical steps that can be described precisely". Such formulas could help the Google team adapt each prompt, Python template and script to tune performance. The paper doesn't say so outright, but it's reasonable to assume that's how they guided the final design.

There are formulas for the problem setup, the Analyzer, the Planner, the Coder, the Verifier and the Router. The Debugger has none, since it simply captures execution problems and fixes them, always yielding working Python. The Finalizer also lacks one, because it is just a prompt to write the final answer to a detailed question, whether that's a table, a number or a summarised report.

The algorithm

The paper's Algorithm 1 gives pseudo-code for the entire flow. Having covered modules and formulas, it reads straightforwardly. In plain words:

Phase 1, data profiling. For each file in the dataset, run the Analyzer and store its description. Everything else rests on this foundation.

Phase 2, iterative planning.

1. The Planner produces the first plan step from the question and data descriptions.
2. The Coder writes code for that step.
3. The Executor runs it. On failure, the Debugger repairs the code and it runs again.
4. The Verifier assesses the cumulative plan against the output. If sufficient, skip to step 6.
5. If insufficient, the Router decides whether to add a step or replace step K. Return to the Planner with the updated context, repeating until the iteration limit.
6. The Finalizer extracts the answer from the accumulated results.

The structure resembles a feedback control loop, the kind found in PID controllers. The Verifier acts as the sensor, the Router as the controller, and the Planner and Coder as actuators. The loop ends when the sensor signals that the job is done.

DS-STAR+ in depth

DS-STAR+ takes an open-ended query and turns it into a structured report. Its official diagram is detailed enough to use as it stands. The flow has three parts.

1. Decomposition comes first. A vague query can't go straight to DS-STAR. DS-STAR+ introduces a Generator agent. Reading both the query and the data descriptions, it writes out a batch of narrow sub-questions. Each has a clear question and a pointer to the relevant data, so DS-STAR can answer it. DS-STAR runs on each one independently and returns answers.
2. A Writer agent then merges the sub-question and answer pairs into a structured report. Its prompt tells it to cite every claim back to the sub-question that grounded it, an attempt to curb hallucination by making each statement traceable to an executed data query.
3. The refinement loop resembles DS-STAR's. The Generator re-reads the current draft along with the data descriptions and produces new sub-questions aimed at whatever is missing or thin.

The notation extends the DS-STAR chain with that decomposition layer. The paper gives the exact formulas, but the algorithm shows more clearly how the loop works and what flows through it.

The prompts behind DS-STAR+

The prompts used for this outer loop are also published. First the sub-question generator, on its first pass:

You are an expert data analysist.
Your task is to write a comprehensive data science report to the given question by using
the files/documents listed below.
In order to do this, you have to first suggest multiple data analysis questions that
should be answered to write the report.

# Given data: {filenames}
{filenames #1}
{summaries #1}
...
{filenames #N}
{summaries #N}

# Question
{question}

# Your task
- Suggest multiple factoid data analysis questions that are required to write the report
really well.
- All the questions should be well-answered using the given data.
- All questions should be answered independently.
- Generate as much as you can.
- Return in valid JSON format:
Questions = {'question': str}
Return: list[Questions]

Then the same agent during refinement:

You are an expert data analysist.
Your task is to complement the given data science report of the given question.
In order to do this, you have to suggest supplementary multiple data analysis questions
that can strengthen to the report.

# Given data: {filenames}
{filenames #1}
{summaries #1}
...
{filenames #N}
{summaries #N}

# Given data science report:
{report}

# Question
{question}

# Your task
- Suggest multiple factoid data analysis questions that are required to complement the
report.
- All questions should contain new information that is not included in the report.
- All the questions should be well-answered using the given data.
- All questions should be answered independently.
- Return in valid JSON format:
Questions = {'question': str}
Return: list[Questions]

Next the Writer, producing the first report:

You are an expert data analysist.
Your task is to write a **comprehensive data science report** to the given question by
using the data and some relevant informations listed below.

# Relevant informations:
{Sub-Question #1}
{Answer #1}
...
{Sub-Question #M_0}
{Answer #M_0}

# Question that you have to write a comprehensive data science report:
{question}

# Your task:
- The report should be grounded to the given relevant informations.
- For the citation, use the Sub-Question number as a citation number which is in 1 - {len(subquestions)}.
- The data science report should be relevant to given question, should be comprehensive,
and should be insightful.
- The data science report should have nice structure, good readability, and should be
professional.
- Write a very comprehensive data science report to the given above question.

And the Writer once more, refining an existing report:

You are an expert data analysist.
Your task is to complement the given data science report of the given question by using
the some relevant informations listed below.

Relevant informations:
{Sub-Question #1}
{Answer #1}
...
{Sub-Question #M_k}
{Answer #M_k}

# Given data science report:
{report}

# Question that you have to write a comprehensive data science report:
{question}

# Your task:
- Do not modify the given report a lot. Just try to add new information.
- The report should be grounded to the given relevant informations.
- Cite with alphabet. For the citation, use the Sub-Question number as a citation
alphabet (e.g., cite with [a] for the Sub-Question 1).
- The data science report should be relevant to given question, should be comprehensive,
and should be insightful.
- The data science report should have nice structure, good readability, and should be
professional.
- Complement the give data science report to the given above question.

For anyone wanting to run the system, a community project, JulesLscx/DS-Star, offers a faithful re-implementation (145 stars and 37 forks when it was checked). It holds dsstar.py with the full DS_STAR_Agent class, prompt.yaml with all prompts as text and provider.py supporting Gemini, OpenAI and Ollama. It isn't an official Google release, and none exists, but it follows the paper accurately.

What the ablations show

Table 4 in the paper asks which component contributes most to DS-STAR's results. The method is ablation by removal: take out one component and measure the drop.

Without the Analyzer, hard-level DABStep accuracy falls from 45.24% to 26.98%, the largest drop for any single removal. With no data descriptions, the Planner writes steps with no grounding in what the data actually contains. Performance still beats the 12.70% non-agentic baseline, so the other parts do contribute, but the Analyzer carries the most weight.

Without the Router (forcing "Add Step" only), accuracy drops on both easy and hard tasks. Being able to replace a flawed step, instead of merely appending to a broken plan, stops errors from compounding. Without that ability the plan, and its errors, just keep accumulating.

Replacing the step-by-step Verifier with a plan-everything-then-execute approach, where code execution success is the only check, performs worse than DS-STAR's method. Checking at the level of individual steps catches plan problems before they spread to the next.

Data scientists will recognise the method as a kind of feature importance study applied to an agent: remove components one at a time and see how far results fall. By that measure the Analyzer ranks first, the Router second, and the step-by-step Verifier beats the full-plan baseline.

Google's research blog adds one more observation. DS-STAR with GPT-5 does better on easy tasks, while DS-STAR with Gemini 2.5 Pro does better on hard ones. Same harness, different model strengths on display. That doesn't argue against strong models. It suggests the harness reveals model strengths rather than averaging them away.

More iterations for harder problems

Hard tasks need an average of 5.6 iterations to reach a sufficient plan, while easy tasks need 3.0. Over half of the easy tasks finish in one round. The system doesn't over-iterate on simple problems: it spends refinement rounds in proportion to the difficulty of the request. An easy question gets one pass, whereas a hard question that spans several datasets and many reasoning steps gets five or six.

That matters in practice, because token cost and latency grow with iteration count. A system that stops early on simple tasks and persists on hard ones is behaving as it should.

Google's example report on feature engineering

Appendix G of the paper includes three sample DS-STAR+ outputs. Report 3 is the more interesting, since it touches feature engineering for machine learning problems. The question asks for a thorough data preparation report aimed at improving how payment processing fees are calculated. It wants the links between merchant traits, transaction attributes and fee structures examined across several datasets, plus a look at data quality problems, feature engineering for the fee calculation, and whether the fee rules actually apply.

The output covers four things:

1. Feature engineering decisions for fee calculation, meaning which fields to join and which rules apply under which conditions.
2. Data quality problems flagged across the datasets, such as missing values, inconsistent encodings and edge cases in rule applicability.
3. A check of whether the fee rules in the manual can really be applied to the given data as they stand.
4. A citation in each section to the sub-question and code execution behind it.

For a working data scientist, a report like this would speed up looking into specific areas of the data considerably.

Limitations, including a Claude Code port

Four limitations are worth naming.

1. No official open-source release. Google hasn't published an implementation. The community re-implementation is faithful and well documented, but the paper's authors don't maintain it.
2. Integration with Claude Code is an open problem. An obvious question is whether DS-STAR patterns could run inside a Claude Code workflow. Claude Code can reach external data sources through Model Context Protocol servers, so a DS-STAR-style harness could in principle be built as a set of skills or agents. No published, production-tested version exists yet.
3. The Analyzer is powerful but basic. At present it generates a Python script and runs it, essentially a programmatic describe(). Tools such as ydata-profiling (formerly pandas-profiling) give far richer data quality summaries, with distribution plots, correlation matrices, missing-value heatmaps and warnings about categorical encodings. Whether richer context from the Analyzer would lift performance further is a real research question the paper leaves open.
4. The name is slightly misleading. Judging by what the agent produces, it looks more like a production data science agent. It's hard to picture it producing machine learning solutions and iterating on them against a target metric. Open-source projects such as Karpathy's autoresearch tool focus on that kind of problem.

Why the harness matters more than the model

DS-STAR works because of its harness. Data is profiled first. Planning then proceeds a single step at a time, with every step checked against what actually ran, and when something fails, that exact step is repaired instead of the system carrying on regardless. Seven focused agents each do one job, each held in check by a short, specific prompt.

The ablation on the Analyzer is the finding that sticks. The largest performance loss comes from removing the step that describes the data files, not the Verifier or the Router. Before planning begins, DS-STAR reads the data and builds a model of it, and every later step rests on that model. Leave it out, and what follows is planning blind.

Anyone aiming for more dependable LLM-based data science workflows can take away this: isn't to add capability to a single agent. Split the work into smaller steps that can be verified, and make sure the first of them is understanding the data.