Optimizing a Terraform Agent With DSPy: Build the Verifier Before the Optimizer

An agent that writes infrastructure code has probably produced a module like this at some point: it looks entirely reasonable, reads like every module a reviewer has ever approved, and then fails terraform validate for a reason nobody would catch by eye.

This gets measured too. IaC-Eval is a benchmark of 458 hand-curated Terraform scenarios, and GPT-4 reportedly manages only 19.36% pass@1 on it, against 86.6% for the same model on tightened Python benchmarks. A newer study of agentic IaC generation reports 14.0% for Qwen2.5-Coder 7B, 41.4% for GPT-4o and 62.4% for GPT-5.4.

Why is infrastructure code so much harder than Python? A correct configuration has to meet four demands at once: valid syntax, a match with the schema of the provider version in use, a dependency graph that holds together, and compliance with the organisation's own policy. Missing one makes the whole thing wrong. Python that's 90% right often still runs, while Terraform that's 90% right doesn't apply.

The research also holds results that should make the standard remedies look doubtful. Few-shot prompting, chain-of-thought and multi-turn interaction each helped only a little. But roughly 79% of policy failures traced back to information missing from the prompt, with model capability not the cause. The model was capable enough. It simply hadn't been told the right things.

That's an optimisation problem. So the plan here is to take a working Terraform agent, build the evaluation needed to know whether it's improving, and then run the full DSPy workflow against that evaluation: program, evaluate, optimise. Everything is runnable. The companion project splits into three folders, one per stage:

agent/       the deepagents Terraform agent, its tools, prompts, and skills
eval/        the verifier, 38 tasks with Rego policies, and the benchmark
optimizer/   the DSPy program, metric, and GEPA compile

The starting agent

Keep this section short, since it isn't the interesting bit and any agent can stand in for it. This one looks up the real schema, writes files that are checked, and validates before answering.

Three tools sit in agent/tools.py. The agent reads the real provider schema ahead of writing. Files it writes are formatted and syntax-checked as they go in. It then validates the full configuration and fixes whatever the compiler objects to, and only afterwards does it reply.

It's built on deepagents, LangChain's framework for agents with tools and skills, though the framework barely matters. The shape is what counts:

from deepagents import create_deep_agent

agent = create_deep_agent(
    model="openrouter:openai/gpt-5.6-luna",
    tools=[provider_schema, write_terraform, validate_config],
    system_prompt=SYSTEM_PROMPT,
    skills=["./agent/skills"],     # SKILL.md — the thing we'll optimize
)

The skills directory merits a closer look, because everything else here optimises it. A skill is a markdown file of instructions that an agent loads next to its system prompt, in a format called Agent Skills that is catching on quickly: Claude, GitHub Copilot and deepagents all read the same SKILL.md layout. It separates concerns neatly. The system prompt remains thin wiring (how to call tools, how to format output) and the skill holds domain knowledge such as Terraform conventions, security defaults and provider quirks. Because the same file runs on several agents, skills are also portable.

Most DSPy write-ups point the optimiser at a program's main prompt. The question tested here is a bit different: can it optimise a skill? Real teams spend most of their manual tuning on the skill, since it holds the expertise and gets edited endlessly in review. In this project it's hand-written, lives in git and is code-reviewed like any other file. And it has never been measured, which is the real problem, and DSPy is designed to address it, because each edit to that file changes production behaviour without any test.

What failure actually looks like

Start with a real artifact from before any fixes. It was produced in the recorded baseline benchmark run, where the agent had no tools and wrote HCL straight into its reply. The request asked for a state bucket using customer-managed encryption:

  encryption {
    kms_key_name = var.encryption_key_name
  }

  lifecycle_rules {
    ...

It looks fine, but running the validator on it tells another story:

$ tofu validate

Error: Missing required argument
  on main.tf line 18, in resource "google_storage_bucket" "terraform_state":
The argument "default_kms_key_name" is required, but no definition was found.

Error: Unsupported argument
  on main.tf line 19, in resource "google_storage_bucket" "terraform_state":
An argument named "kms_key_name" is not expected here.

Error: Unsupported block type
  on main.tf line 22, in resource "google_storage_bucket" "terraform_state":
Blocks of type "lifecycle_rules" are not expected here. Did you mean
"lifecycle_rule"?

Three errors came out of a single resource. The required argument is default_kms_key_name, while kms_key_name belongs to other Google resources, which is just why it feels correct. A block name had also been pluralised into something that doesn't exist.

When the baseline failures were sorted into types, 80% of the validate errors were invented schema: arguments, blocks or attributes that look like genuine HCL but don't exist on the resource in question. The google provider has about a thousand resource types and changes with every release, so the model is reciting a versioned API from memory, and memory yields plausible fictions.

The three tools target that diagnosis directly. On their own they added 27.7 points (the pass rate rose from 41.7% to 69.4% on the suite described below). Formatting errors fell to zero because write_terraform runs fmt rather than expecting the model to manage whitespace. Schema errors fell to zero because the agent could at last see what the compiler sees. What was left consisted almost entirely of intent, whether the agent built what was requested, and that is where DSPy comes in.

Here's the same task run through the tool-enabled agent as it works today:

$ uv run python -m agent.run --task eval/tasks/backend-var-interpolation.json

task      backend-var-interpolation (opentofu)
model     openai/gpt-5.6-luna  engine=deepagents
files     main.tf, variables.tf
tools     {'write_terraform': 2, 'validate_config': 1, 'validate_pass': 1}
elapsed   56602ms

There were two writes and one validation, and the agent answers only once the compiler is happy. The task file and the opentofu target come from the evaluation suite, which is where things head next.

A scorer before a framework

Judging whether any of it helps means answering "is the output good?" with something firmer than a feeling. Most fields use an LLM judge, which pits one model's opinion against another's with no ground truth, and then the judge itself needs checking.

Terraform doesn't have that problem, because the toolchain already running in CI is the scorer:

• tofu fmt -check catches formatting.
• tofu validate finds syntax, schema and reference errors.
• tflint flags deprecated arguments and invalid machine types.
• tofu test with mock_provider checks semantics without any cloud credentials.
• conftest with OPA Rego checks intent, asking whether the agent built what was requested.

Those five deterministic gates are implemented in eval/verifier/run.py. No judge is involved, evaluation costs nothing in API calls, and no step can produce a billable resource, because the project contains no apply anywhere, on purpose.

Most people skip the final gate, though it counts for the most. validate says the configuration compiles. Rego says the bucket really has encryption switched on, which is what was asked and what makes it correct rather than merely valid:

package main
import rego.v1

deny contains "bucket must use a customer-managed KMS key" if {
    some name
    bucket := input.resource.google_storage_bucket[name][_]
    not bucket.encryption
}

A team that already enforces policy through OPA likely has many such policies in Terraform Cloud or its CI pipeline, and they can be reused unchanged. It's shift-left, applied to agents. Terraform Cloud does turn down a bad plan at apply time, yet by then the code exists, a reviewer has read it and a run is queued. Running the same Rego when the code is generated finds the violation before all that effort is spent.

Test the tester. This is the step most worth copying. A policy that approves everything is worse than none, because it gives every agent version the same score and silently reports no change. Each policy in eval/tasks/policy/ therefore ships with one compliant fixture and one broken fixture, and a script checks both directions:

$ cd eval/tasks && ./check_policies.sh
  ...
  ✓ vpc-subnet-firewall              good=0 bad=2

✅ 38/38 policies verified in both directions

Thirty-eight tasks were written this way. They cover security defaults, wiring between resources, breadth across provider schemas, dialect targeting, restraint (does the agent add things nobody wanted) and file layout. Each pairs a prompt with a Rego policy that captures its intent. Dialect gets special treatment: each task states whether it targets terraform or opentofu, since the two have drifted apart (OpenTofu offers state encryption and provider for_each, Terraform offers Stacks), so an agent that silently targets the wrong one fails validation.

The diagram of the five gates also shows a measurement boundary the agent can't cross, and it's easy to overstate what that means. In production the agent can, and probably should, get an OPA tool so it checks policies before answering. That's the same shift-left argument, and it ships better code.

The boundary has a different purpose. Here those same policies are the yardstick, used to compare skills, models and optimiser runs. If the agent could iterate against the very policy that grades the experiment, all variants would drift toward passing and the comparison would say nothing. This is the held-out test-set principle, applied to tools. The rule: the agent may see any check at all except the one currently grading the experiment. If the production agent will ship with the policy tool, evaluate that configuration too, grading with held-back policies or counting the repair iterations needed instead of whether it passes.

Adopting DSPy

DSPy describes itself as a framework for programming language models rather than prompting them. Its documentation organises the workflow into three phases, programming, evaluation and optimisation, and the order is itself the lesson: the optimiser is the last thing touched, and everything before it is what makes the optimiser trustworthy. The optimiser consumes exactly three things: a program, a metric and data.

All of the DSPy work sits under optimizer/, and setup relies on uv:

uv add dspy
uv sync

Phase 1: programming

A signature is a typed contract for one LM call, made of inputs, outputs and a docstring describing the task. This one lives in optimizer/harness.py:

import dspy

class AuthorConfiguration(dspy.Signature):
    """Write a complete, valid Infrastructure-as-Code configuration."""

    request: str = dspy.InputField(
        desc="What the user wants built, in natural language."
    )
    target: str = dspy.InputField(
        desc="Which dialect to target: 'terraform' or 'opentofu'. These have "
             "diverged — code targeting the wrong one will fail validation."
    )
    config: str = dspy.OutputField(
        desc="The complete configuration as fenced HCL blocks. Start each block "
             "with a comment naming its file, e.g. '# main.tf'."
    )

The omissions are the point. There's no "you are an expert Terraform engineer", no worked examples and no pleading about format. Field names and descriptions carry the meaning, and the optimiser takes care of phrasing.

Anyone who finds this magical can dispel it with one line:

dspy.inspect_history(n=1)

It prints the actual prompt the adapter built from the signature. Real output from this one looks like this:

System message:

Your input fields are:
1. `request` (str): What the user wants built, in natural language.
2. `target` (str): Which dialect to target: 'terraform' or 'opentofu'. ...
Your output fields are:
1. `config` (str): The complete configuration as fenced HCL blocks. ...

[[ ## request ## ]]
{request}

[[ ## target ## ]]
{target}

[[ ## config ## ]]
{config}

In adhering to this structure, your objective is:
        Write a complete, valid Infrastructure-as-Code configuration ...

Nothing is hidden. It's a structured template produced from the signature, which is exactly why an optimiser can safely rewrite it, and why it should be the first place to look when behaviour is odd.

A module applies a technique to a signature, and the two vary independently. One signature can run as a single shot, with reasoning first, or as the driver of a tool loop:

generate = dspy.Predict(AuthorConfiguration)             # one shot
generate = dspy.ChainOfThought(AuthorConfiguration)      # reason first
generate = dspy.ReAct(AuthorConfiguration, tools=[...])  # run a tool loop

This agent uses tools, so it is mirrored with dspy.ReAct and given exactly the tool functions the deployed agent calls, imported directly from the agent package. The code is in optimizer/harness_deepagents.py:

from agent.tools import make_tools            # the deployed agent's tools

class TerraformAuthoringAgent(dspy.Module):
    def __init__(self, seed_instruction: str):
        super().__init__()
        # Seed the SIGNATURE before constructing ReAct, so DSPy appends its
        # tool protocol to your instruction instead of replacing it.
        seeded = AuthorConfiguration.with_instructions(seed_instruction)
        lc_tools = make_tools(work_dir, binary="terraform", stats={})
        self.react = dspy.ReAct(
            seeded,
            tools=[dspy.Tool(t.func, name=t.name, desc=t.description)
                   for t in lc_tools.values()],
            max_iters=10,
        )

    def forward(self, request: str, target: str = "terraform"):
        return self.react(request=request, target=target)

The most important design decision in the setup is this: optimise the same program that gets deployed. An early experiment optimised a one-shot generator and then shipped the output into a tool-using agent, which made every number meaningless, since the instructions had been tuned for failures the real agent no longer produced.

The seed instruction is the existing SKILL.md, so there is no fresh start, and the real question is whether the hand-written text can be bettered. DSPy doesn't know that the string is a skill instead of a main prompt, because it sees only the signature's instruction text. This is what makes a skill experiment workable: the result is saved straight back in SKILL.md form and works for any agent that reads skills.

Phase 2: evaluation

The data type in DSPy is dspy.Example, a dict aware of which fields are inputs. The loader in optimizer/dataset.py converts the suite directly:

dataset = [
    dspy.Example(
        task_id=t["id"],
        request=t["prompt"],
        target=t["target"],
        rego=t["rego"],                # path to this task's policy
    ).with_inputs("request", "target")
    for t in tasks
]

Fields not flagged as inputs travel along as metadata. The metric uses rego to find the matching policy, and the model never sees it.

How the examples get picked is worth spelling out, because the dataset shapes everything the optimiser can learn. They weren't scraped from a benchmark. They imitate how real tickets read, for example a request for a GCS state bucket encrypted with a customer-managed key, or a service account allowed into one bucket and nothing beyond it. Coverage then spans the core resources a platform team handles: storage, networking, IAM, SQL and Pub/Sub, plus families aimed at failure modes already seen: schema-heavy resources where models invent arguments, restraint tasks checking that the agent doesn't add resources nobody asked for, and the dialect tasks above. A few are deliberately hard, since a suite that passes everything on day one can't show improvement.

Thirty-eight examples sounds small, and it is. The aim is coverage per example, since each carries its own policy, and honest splits. Split before optimising and keep the test set sacred: train is what the optimiser learns from, validation is how it picks candidates, and test gets scored once, at the end. Most prompt engineering skips this discipline, and skipping it is how people convince themselves a change helped when it didn't. It happened twice in this project, and both times a win on a small validation sample reversed once the sample grew.

The metric is the place where the verifier plugs in, and it is one function doing three jobs (optimizer/metric.py):

def verify_metric(example, prediction, trace=None, pred_name=None, pred_trace=None):
    """Score with the SAME verifier that produces our benchmark numbers."""
    result = verify(
        config_dir=materialise(prediction.config),
        policy_dir=example.rego,
        target=example.target,
    )

    # Job 1 — bootstrapping (trace is set): a strict bool. Only outputs that
    # FULLY pass may become worked examples.
    if trace is not None:
        return result.passed

    # Job 2 — reflective optimization (pred_name is set): score AND feedback.
    # GEPA reads the text to understand *why* a candidate failed.
    if pred_name is not None:
        return dspy.Prediction(score=result.score, feedback=format_failures(result))

    # Job 3 — plain evaluation: a float.
    return result.score

Keep it thin. If the metric reimplements any scoring logic, the number being optimised and the number being published drift apart, and then nothing is being measured.

The feedback branch is where this domain stands out. Reflective optimisers take text that explains a failure, and here that text is free and exact: genuine compiler errors and genuine policy denials, like those shown earlier. Fields that depend on an LLM judge pay for each evaluation and get far worse feedback, and then must validate the judge as well.

Measure a baseline before anything else, using dspy.Evaluate:

evaluate = dspy.Evaluate(devset=valset, metric=verify_metric,
                         num_threads=8, display_table=True)
evaluate(program)

Output from a 3-example smoke run looks like this, and the full suite takes longer:

Average Metric: 0.59 / 3 (19.6%): 100%|██████████| 3/3 [01:16<00:00, 25.60s/it]
INFO dspy.evaluate.evaluate: Average Metric: 0.5882 / 3 (19.6%)
WARNING dspy.evaluate.evaluate: Skipping table display since `pandas` is not installed.

Two lessons come from that snippet. Each "example" is a full agent episode, not a single API call, so budget accordingly. And display_table quietly needs pandas, so install it for the per-example table.

If one habit comes out of this, make it that nothing gets optimised without a baseline. That number is the only thing that makes the optimiser's output meaningful.

Phase 3: optimisation

Different optimisers tune different things:

• BootstrapFewShot synthesises worked examples (demos).
• MIPROv2 adjusts instructions and demos with Bayesian search.
• GEPA tunes instructions by reflecting on failure traces.

Instruction tuning is the right lever for schema-constrained code, since the agentic IaC study found it outperformed injecting demonstrations, precisely because demos don't cure schema hallucination. GEPA is therefore the pick, driven from optimizer/compile.py:

gepa = dspy.GEPA(
    metric=verify_metric,
    max_metric_calls=150,
    reflection_lm=dspy.LM("openrouter/openai/gpt-5.6-luna", max_tokens=16000),
    num_threads=8,
    track_stats=True,
)

compiled = gepa.compile(student=program, trainset=trainset, valset=valset)
compiled.save("compiled_state.json", save_program=False)

The two models do different work. The task model writes Terraform and is what ships; the reflection model studies failures and drafts candidate instructions. It runs about 40 times to the task model's roughly 900, so spending on a stronger model there is affordable.

Some hard-won practical notes:

• Set num_threads. In one run it was forgotten and compiling took 69 minutes; measuring afterwards showed the machine idle for 94% of the time, waiting on API calls made one after another.
• "150 metric calls" means 150 agent episodes, which with a tool loop is roughly 900 LM calls.
• Always call save() after compiling. With save_program=False the output is plain JSON: the optimised instruction text plus any bootstrapped demos. The same JSON is a way out of DSPy entirely, since roughly twenty lines (optimizer/to_skill_md.py) convert it again into the SKILL.md file the agent consumes. At runtime, DSPy is never imported. The optimiser is just a build step, and what it produces arrives in a pull request as a diff that a person can read and reject.

GEPA also finds real things. On a weaker model the agent kept abandoning its tool loop: one provider_schema call, then it stopped without writing anything. GEPA was never told that. It read the failure traces and wrote this into the optimised instruction: don't finish merely because a file was written, since a successful write proves only that the HCL parses, and validate_config is required to catch provider-schema errors and invalid references. That's reflection on real execution traces, producing guidance that would have taken hours of debugging to phrase.

The workflow keeps paying after optimisation

Two more pieces of the same machinery reuse the metric that's already there.

The metric can also run at inference time. DSPy's BestOfN and Refine wrap any module, run it as many as N times and return the first result whose reward reaches a threshold:

robust = dspy.Refine(module=program, N=3, reward_fn=verify_metric, threshold=1.0)

The pattern should look familiar, since the agent's validate-and-repair loop applies the same idea with the compiler as reward function. The evidence ranks it as the strongest lever there is: in the IaC study, iterative repair with verifier feedback took Qwen 7B from 14.0% to 62.9%, better than every prompting technique tried. In whatever form, supply the model with its verifier at runtime.

Model swaps become a one-line change, and now they're measurable:

program.set_lm(dspy.LM("openrouter/qwen/qwen3-coder-30b-a3b-instruct"))
evaluate(program)

This mattered more than expected. A mid-project switch to a cheaper model dropped quality from 83.3% to 37.5%, with nearly half the runs abandoning the tool loop and producing no files at all. Nothing in a manual workflow surfaces that, and the first sign would come from a customer. Twelve minutes of running the eval suite exposed it.

The honest result

For this agent, the compiled instruction failed to outperform the hand-written one.

The comparison ran nine times over two agent implementations and two models, including a held-out test set on which both scored the same 92.9%. That figure is for the stronger model, while the earlier 41.7%-to-69.4% tool numbers are for the cheaper one on the same suite. Twice a result seemed a clear win, was written up, and then flipped on a bigger sample.

That's the most valuable thing learned, and it doesn't show that DSPy fails. A specific and plausible reason may explain why it didn't apply. GEPA's published gains come from modular workflows, pipelines whose steps each use a different prompt, so the optimiser can blame one module for a failure and repair it. An agent uses one prompt at every step. Benjamin Anderson's critique argues that this removes the very locality GEPA relies on.

The practical guidance follows:

• For a straight pipeline, the published gains probably carry over.
• For an agent loop, run your own suite before assuming they do.
• In both cases, route verifier feedback into the agent itself; that has the best evidence behind it, in published work and in these numbers.

And that test is possible, because the suite exists.

Where this leaves things

Language models find infrastructure as code difficult. GPT-4's 19.36% on IaC-Eval is no prompt problem that wishful thinking will remove, and the familiar tricks barely help. Still, most policy failures arise from missing information, not missing ability, so real headroom exists, and it sits in instructions and tooling rather than in weights.

The workflow that earned its keep is the one DSPy's own documentation teaches, in the order it teaches it. For programming: signatures for the contract, modules that mirror what gets deployed, and inspect_history when there's doubt about what's being sent. For evaluation: examples with explicit inputs, a thin metric that doubles as optimiser feedback, a baseline before any change and a test set touched once. For optimisation, only then, with the same metric, saving text that can be reviewed in a pull request.

Even though the optimiser didn't beat the hand-written skill, adopting DSPy forced the definition of "correct" to be written down as executable code. That artifact kept earning its place. It flagged a 46-point regression after a model swap, revealed that the skill was loaded only 25% of the time (fixed with one rewritten sentence, after which the load rate reached 100%), and twice stopped a noise result from being published as a win.

None of that strictly needs DSPy, and the verifier here imports none of it. Where DSPy helps is in forcing a metric to exist and be reused, so one definition drives the regression suite, the model comparisons, the inference-time retries and the optimiser. Teams that already have the discipline may not need the framework, and those that don't can get it from there. Either way, stop shipping prompt edits that can't be measured.