Prompt Engineering and Structured Output in .NET: OpenAI, Azure, Anthropic, Gemini

A support bot that calls GPT-4o straight from an ASP.NET Core controller can win a demo. Then the enterprise customer asks for regional data residency, security wants Claude for long documents, and finance wants to know why each question costs ten times the estimate. Every one of those requests pulls you toward a different provider, or toward an abstraction that lets you swap them.

OpenAI, Azure OpenAI, Anthropic and Gemini all sell the same basic thing: managed chat, tool calling and embeddings without hosting GPUs. They differ in packaging, regions, compliance story, SDK quality and how they bill. For a .NET service that will live for years, those differences matter more than whichever model topped a leaderboard last week.

Which provider fits which job

• OpenAI platform: the fastest route to the newest models. Fine for products that are already comfortable with OpenAI's data boundaries.
• Azure OpenAI: when the workload has to live inside an Azure tenant with enterprise identity, networking and regional controls.
• Anthropic: strong long-context work and careful instruction following. Evaluate it for document-heavy agents.
• Gemini: stacks aligned with Google Cloud, multimodal strengths, and products that sit close to Workspace.

Don't go multi-provider by default. With one team, one region and no compliance driver, a single provider means fewer moving parts. Every abstraction charges a tax.

SDK maturity and the packages to install

What you're judging here is official client libraries, idiomatic async patterns and how often breaking changes land. An SDK that fights HttpClient lifetime and dependency injection will cost you more than the token bill. Expect first-party or well-supported community packages for all four, but check the current NuGet versions and auth flows (API key versus Entra ID) before you standardize on anything.

// Official OpenAI SDK
dotnet add package OpenAI

// Azure AI Foundry (Azure OpenAI)
dotnet add package Azure.AI.OpenAI

// Anthropic Claude (official, beta as of 2026)
dotnet add package Anthropic

// Gemini via Google Cloud Vertex AI (IChatClient-compatible, January 2026)
dotnet add package Google.Cloud.VertexAI.Extensions --prerelease

// Abstraction layer (ships with all provider adapters)
dotnet add package Microsoft.Extensions.AI

Microsoft.Extensions.AI is the piece that ties them together: each SDK exposes an IChatClient, so the rest of the app can stay provider-neutral.

Pricing mechanics and model routing

Every provider charges for tokens, but the details move the total: input and output tokens are priced differently, cached tokens are cheaper, tooling can carry a premium, and some plans have minimums. Long context and tool-heavy agents skew the bill fast. Azure bills through Azure metering, while the other three bill a provider account.

Finance will build a unit-economics model, so your architecture has to emit usage numbers they trust. Put every completion call behind a meter that logs model, tokens and feature flag. Budgets belong in code and in cloud billing alerts. Routing by task complexity is the cheapest win:

// Partial example — model routing based on task complexity
public class ModelRouter
{
    // Route cheap, simple tasks to the nano model
    // Route complex multi-step reasoning to the full model
    public string SelectModel(TaskComplexity complexity) => complexity switch
    {
        TaskComplexity.Classification  => "gpt-5.4-nano",   // ~20x cheaper than full
        TaskComplexity.Conversational  => "gpt-5.4-mini",   // ~5x cheaper
        TaskComplexity.Reasoning       => "gpt-5.4",         // full capability
        _                              => "gpt-5.4-mini"
    };
}

Enterprise fit: identity, network and residency

Security questionnaires kill projects that only optimized for demo latency. The questions are about identity, network isolation, data retention and regional availability.

Azure OpenAI inherits Azure RBAC, Private Link and region choice. OpenAI and Anthropic offer organization controls and zero-retention options, with caveats you need to read. Gemini follows Google Cloud IAM and VPC patterns.

On auth inside ASP.NET Core, prefer managed identity for Azure OpenAI. For API-key providers, load secrets from Key Vault at startup and rotate them without redeploying prompts. Never bind provider keys to user JWTs.

Legal review has to cover training-use opt-out, subprocessors and DPA language for every provider you enable, not only the primary one. Multi-provider means multi-contract.

Structured output with JSON Schema

Support for JSON Schema and constrained decoding varies between vendors. Prefer providers that can enforce a schema for the tool arguments and responses your C# deserializers already trust, and fall back to validate-and-retry where enforcement is weak. With Azure OpenAI you can make the schema strict:

// Full example — Azure OpenAI structured output with strict schema enforcement
using Azure.AI.OpenAI;
using OpenAI.Chat;
using System.ComponentModel;
using System.Text.Json;

// Define the schema type — only JSON Schema primitives are supported
public class SupportTicketAnalysis
{
    public string Category { get; set; } = string.Empty;        // e.g. "billing", "technical"
    public string Priority { get; set; } = string.Empty;        // "low", "medium", "high", "critical"

    [Description("One to two sentence summary of the issue")]
    public string Summary { get; set; } = string.Empty;

    public string[] RequiredTeams { get; set; } = [];           // which teams must be looped in

    [Description("Estimated resolution time as a human-readable string, e.g. '2 business days'")]
    public string EstimatedResolution { get; set; } = string.Empty;
}

// Build the client — structured outputs require API version 2024-08-01-preview or later
var azureClient = new AzureOpenAIClient(
    new Uri(Environment.GetEnvironmentVariable("AZURE_OPENAI_ENDPOINT")!),
    new AzureKeyCredential(Environment.GetEnvironmentVariable("AZURE_OPENAI_KEY")!),
    new AzureOpenAIClientOptions(AzureOpenAIClientOptions.ServiceVersion.V2024_08_01_Preview));

ChatClient chatClient = azureClient.GetChatClient("gpt-5.4-mini-deployment");

// Configure the request to enforce the schema strictly
var options = new ChatCompletionOptions
{
    ResponseFormat = ChatResponseFormat.CreateJsonSchemaFormat<SupportTicketAnalysis>(
        "support_ticket_analysis",  // snake_case — spaces and hyphens cause 400 errors
        strict: true)               // without this, schema compliance is best-effort only
};

List<ChatMessage> messages =
[
    new SystemChatMessage("You are a support ticket classifier. Analyze the ticket and return a structured assessment."),
    new UserChatMessage("User reports: 'My invoice shows $999 but the contract says $299. Need this fixed immediately.'")
];

// If structured outputs fail with a 400, the schema has unsupported types — check for DateTime, Uri, TimeSpan
ClientResult<ChatCompletion> result = await chatClient.CompleteChatAsync(messages, options);

// JsonSerializer.Deserialize will always succeed if the API call succeeded
SupportTicketAnalysis analysis = JsonSerializer.Deserialize<SupportTicketAnalysis>(
    result.Value.Content.Text)!;

Console.WriteLine(quot;Category: {analysis.Category}");
Console.WriteLine(quot;Priority: {analysis.Priority}");
Console.WriteLine(quot;Summary: {analysis.Summary}");

Two details in that sample cause most 400 errors: schema names need snake_case, and types like DateTime, Uri or TimeSpan aren't JSON Schema primitives. Without strict: true, compliance is best effort only.

Tool calling, embeddings and RAG

All four support tool or function calling, each with a different payload shape. Normalize to an internal ToolCall DTO at the boundary so orchestration code doesn't fork per vendor.

Choosing a chat vendor doesn't mean choosing its embeddings. Pin the embedding model and dimension in config, and don't switch them silently when you swap chat providers. Cache embeddings and deterministic tool results aggressively, but don't cache raw chat completions that contain user PII unless a retention policy covers them.

A thin .NET abstraction with IChatClient

Define a small provider interface with streaming and non-streaming methods, map each SDK behind an adapter, and inject it through DI. Keep prompts and tools outside the adapters. Skip the "god" frameworks that hide the retries and timeouts you'll need to tune. With IChatClient, registration reads the provider from config and wraps everything in OpenTelemetry:

// Partial example — Program.cs provider registration with IChatClient
using Microsoft.Extensions.AI;
using Azure.AI.OpenAI;
using OpenAI;
using Anthropic;
using Google.Cloud.VertexAI.Extensions;

var builder = WebApplication.CreateBuilder(args);
var config = builder.Configuration;

// Read which provider to use from configuration — no code change to switch
string provider = config["AI:Provider"]!; // "azure", "openai", "anthropic", "gemini"

builder.Services.AddSingleton<IChatClient>(sp =>
{
    // Return the appropriate IChatClient implementation based on config
    return provider switch
    {
        "azure" => new AzureOpenAIClient(
                        new Uri(config["AI:Azure:Endpoint"]!),
                        new Azure.AzureKeyCredential(config["AI:Azure:Key"]!))
                    .GetChatClient(config["AI:Azure:Deployment"]!)
                    .AsIChatClient(), // extension method from Azure.AI.OpenAI

        "openai" => new OpenAIClient(config["AI:OpenAI:Key"]!)
                    .GetChatClient(config["AI:OpenAI:Model"]!)
                    .AsIChatClient(),

        "anthropic" => new AnthropicClient(config["AI:Anthropic:Key"]!)
                    .AsIChatClient(), // Anthropic SDK v10+ implements IChatClient

        "gemini" => new VertexAIChatClient(
                        projectId: config["AI:Gemini:ProjectId"]!,
                        location: config["AI:Gemini:Location"]!,
                        modelId: config["AI:Gemini:Model"]!),

        _ => throw new InvalidOperationException(quot;Unknown AI provider: {provider}")
    };
});

// Add OpenTelemetry instrumentation via pipeline decorator — applies to all providers
builder.Services.AddSingleton(sp =>
{
    var inner = sp.GetRequiredService<IChatClient>();
    return new ChatClientBuilder(inner)
        .UseOpenTelemetry(loggerFactory: sp.GetRequiredService<ILoggerFactory>(),
                          sourceName: "YourApp.AI")
        .Build();
});

Feature flags should pick the provider per tenant or per skill. A single static client hardcoded in Program.cs is how residency exceptions turn into incidents.

Streaming needs an explicit contract in the abstraction too: cancellation tokens, backpressure, and how partial tool calls reach callers. Providers differ enough here that "just await the string" leaks vendor semantics into your controllers.

A working proof of concept

The whole idea fits in a small Web API. Start the project and add the packages:

dotnet new webapi -n AiProviderPoc --no-openapi
cd AiProviderPoc

dotnet add package Microsoft.Extensions.AI
dotnet add package OpenAI
dotnet add package Azure.AI.OpenAI
dotnet add package Anthropic
dotnet add package Google.Cloud.VertexAI.Extensions --prerelease

The prompt contract is shared by every provider. The output type sticks to JSON Schema primitives, and the descriptions double as guidance for the model:

// PromptContracts.cs — shared across all providers
using System.ComponentModel;

namespace AiProviderPoc;

// The input to every analysis request
public record AnalysisRequest(string TicketText, string? Context = null);

// The structured output — must use only JSON Schema primitives (string, int, bool, arrays)
public class TicketAnalysis
{
    [Description("Category of issue: 'billing', 'technical', 'account', 'feature-request'")]
    public string Category { get; set; } = string.Empty;

    [Description("Severity: 'low', 'medium', 'high', 'critical'")]
    public string Severity { get; set; } = string.Empty;

    [Description("One to two sentence summary of the customer's issue")]
    public string Summary { get; set; } = string.Empty;

    [Description("Suggested action for the support agent, one sentence")]
    public string SuggestedAction { get; set; } = string.Empty;

    public bool RequiresEscalation { get; set; }

    [Description("Estimated time to resolve as human-readable string, e.g. '30 minutes' or '2 business days'")]
    public string EstimatedResolution { get; set; } = string.Empty;
}

The service owns the system prompt as a stable, versioned contract and depends only on IChatClient. It uses provider-neutral JSON mode; for strict schema enforcement on Azure or OpenAI, use the direct SDK approach shown earlier.

// TicketAnalysisService.cs
using Microsoft.Extensions.AI;
using System.Text.Json;

namespace AiProviderPoc;

public class TicketAnalysisService(IChatClient chatClient)
{
    // The system prompt is the stable contract — versioned and centralized here
    private const string SystemPrompt =
        """
        You are a support ticket classifier for an enterprise software product.
        Analyze the provided support ticket and return a structured assessment.
        Be concise and accurate. Do not invent information not present in the ticket.
        """;

    public async Task<TicketAnalysis> AnalyzeAsync(AnalysisRequest request, CancellationToken ct = default)
    {
        // Build the message list — system prompt is fixed, user content is the ticket
        var messages = new List<ChatMessage>
        {
            new(ChatRole.System, SystemPrompt),
            new(ChatRole.User, request.TicketText)
        };

        // Use JSON mode via ChatOptions — provider-agnostic structured output
        // For strict schema enforcement with Azure/OpenAI, use the SDK-direct approach instead
        var options = new ChatOptions
        {
            ResponseFormat = ChatResponseFormat.Json
        };

        ChatResponse response = await chatClient.GetResponseAsync(messages, options, ct);

        // Deserialize — IChatClient normalizes the response shape across all providers
        return JsonSerializer.Deserialize<TicketAnalysis>(response.Text,
            new JsonSerializerOptions { PropertyNameCaseInsensitive = true })
            ?? throw new InvalidOperationException("Model returned unparseable JSON");
    }
}

Program.cs registers one client per provider branch and exposes a single endpoint that never learns which vendor answered:

using AiProviderPoc.Api;
using Anthropic;
using Azure.AI.OpenAI;
using Google.GenAI;
using Microsoft.Extensions.AI;
using OpenAI;
using System.ClientModel;          // ApiKeyCredential

var builder = WebApplication.CreateBuilder(args);
var config = builder.Configuration;

// Read provider selection from config — change "AI:Provider" to switch providers
string provider = config["AI:Provider"]
    ?? throw new InvalidOperationException("AI:Provider must be set in appsettings");

// Register IChatClient — each branch returns a different SDK's implementation
builder.Services.AddSingleton<IChatClient>(_ =>
{
    return provider.ToLowerInvariant() switch
    {
        "azure" =>
            // Azure AI Foundry: needs endpoint + key + deployment name
            new AzureOpenAIClient(
                new Uri(config["AI:Azure:Endpoint"]!),
                new ApiKeyCredential(config["AI:Azure:Key"]!))
            .GetChatClient(config["AI:Azure:Deployment"]!)
            .AsIChatClient(),

        "openai" =>
            // OpenAI direct: needs API key + model name
            new OpenAIClient(config["AI:OpenAI:Key"]!)
            .GetChatClient(config["AI:OpenAI:Model"]!)
            .AsIChatClient(),

        "anthropic" =>
            // AnthropicClient expects a ClientOptions instance (not a string API key).
            // Create a ClientOptions and pass it to the constructor.
            new AnthropicClient(
                new Anthropic.Core.ClientOptions
                {
                    ApiKey = config["AI:Anthropic:Key"]!
                })
            .AsIChatClient(),

        "gemini" =>
            // Google.GenAI: named params on Client constructor, no ClientConfig class needed.
            // Uses GOOGLE_APPLICATION_CREDENTIALS / ADC for auth; no API key required for Vertex AI.
            new Client(
                project: config["AI:Gemini:ProjectId"]!,
                location: config["AI:Gemini:Location"]!,
                vertexAI: true)
            .AsIChatClient(config["AI:Gemini:Model"]!),

        _ => throw new InvalidOperationException(quot;Unknown provider: '{provider}'. Valid: azure, openai, anthropic, gemini")
    };
});

// Register the service — depends only on IChatClient
builder.Services.AddScoped<TicketAnalysisService>();

var app = builder.Build();

// Single endpoint — provider is invisible here
app.MapPost("/analyze", async (AnalysisRequest request, TicketAnalysisService service, CancellationToken ct) =>
{
    var analysis = await service.AnalyzeAsync(request, ct);
    return Results.Ok(analysis);
});

app.Run();

Switching providers is a config change in appsettings.json:

{
  "AI": {
    "Provider": "azure",
    "Azure": {
      "Endpoint": "https://your-resource.openai.azure.com/",
      "Key": "your-azure-key",
      "Deployment": "gpt-5.4-mini-deployment"
    },
    "OpenAI": {
      "Key": "sk-...",
      "Model": "gpt-4o-mini"
    },
    "Anthropic": {
      "Key": "sk-ant-..."
    },
    "Gemini": {
      "ProjectId": "your-gcp-project",
      "Location": "us-central1",
      "Model": "gemini-2.0-flash"
    }
  }
}

Run it and post a ticket:

dotnet run

curl -X POST http://localhost:5000/analyze \
  -H "Content-Type: application/json" \
  -d '{"ticketText": "My subscription renews tomorrow but I was charged twice this month. Account #A-1234."}'

The response comes back in the same shape whichever provider is configured:

{
  "category": "billing",
  "severity": "high",
  "summary": "Customer was charged twice for their subscription and is seeking a refund before their next renewal date.",
  "suggestedAction": "Review billing history for account A-1234 and initiate a refund for the duplicate charge.",
  "requiresEscalation": false,
  "estimatedResolution": "1 business day"
}

Prompt engineering, evals and fair comparisons

When you compare quality, freeze prompts and temperature and vary only provider and model. Otherwise you're comparing prompt engineering luck, not vendors. Keep prompt templates versioned and A/B tested independently of the provider adapter, or every model swap invalidates your prompt experiments.

Emit OpenTelemetry spans around provider calls with provider, model, token counts and finish reason as attributes. Pair those online metrics with offline eval sets so moving to a cheaper model is a decision backed by evidence rather than hope.

Latency budgets differ as well. Some providers stream the first token faster; others win on long completions. Measure both time to first token and end-to-end time for your workload before you crown a default.

Running more than one provider in production

• Canary: keep one golden prompt per environment that asserts schema-valid JSON and a latency SLO, and fail deploys that break it for the primary provider.
• Content filters: Azure content filters and other safety layers return different error shapes. Map them to your domain exceptions once.
• Failover: decide which errors may be retried on a secondary provider and which must fail closed, such as authorization errors and policy refusals. Blind failover can break residency promises.
• On-call: train people on each enabled provider's billing dashboard. A token spike at 2am is easier to stop when the runbook names the exact portal blade.
• Deprovisioning: disable keys, remove app registrations, purge eval datasets that hold customer text, and archive cost reports.

A quick decision checklist

Residency required? Start with Azure OpenAI. Long documents? Benchmark Anthropic. Already on GCP? Evaluate Gemini. Need the newest OpenAI models the day they ship? Use the OpenAI platform. Need two of the above? Build the thin adapter and usage metering first.

Provider choice is an enterprise architecture decision with SDK and cost consequences. Compare on residency, identity, pricing transparency, structured output and operational fit.