Observability for Enterprise AI Agents With Langfuse and OCI

Logs, metrics and traces won't tell you why an enterprise AI agent gave a bad answer. To understand these systems you also need to see the prompts, the context handed to the model, the tool calls, token usage, response times and the quality of the output. This piece explains why generative AI needs a different approach to observability, then shows a concrete setup using Langfuse that plugs into OCI Enterprise AI Agents through its compatibility with the OpenAI Responses API.

Why this matters

"Agent" gets used for any workflow that calls one or more large language models to produce a result. That blurs a real distinction. A true agent brings autonomy, tool use and context management, and with them far more operational complexity.

A second problem is more pressing in enterprises. Attention flows to the "magic" of AI, while the work needed to run these systems reliably, under control and at good quality gets neglected. Observability sits near the top of that list. For a complex GenAI agent it isn't a technical nicety. It's what lets you understand behavior, analyze errors, judge answer quality and keep cost and latency in check.

The first half below defines observability and what it takes on for generative AI. The second half walks through an open-source setup that's easy to run on OCI, with examples of what it collects and why that data helps.

What observability means

Observability is a property of complex systems. Put simply, a system is observable if you can work out its internal behavior from the signals it emits and that you're able to collect and analyze. More formally, two different internal states have to be distinguishable from the outward signals. What the system emits must reliably reveal what's going on inside.

Two consequences follow. Observability matters most where there are many parts, dependencies and interactions. And watching from the outside isn't enough, because the system has to be designed and instrumented to produce useful information. Running something like an enterprise AI agent properly means collecting telemetry that exposes its state, behavior, anomalies and performance.

That's where observability differs from classic monitoring. Monitoring answers questions you thought of in advance, such as how much CPU a service uses or how slow a request was. Observability must also let you investigate surprises, the problems and questions nobody anticipated at design time, often called unknown unknowns.

Observability before generative AI

"Traditional systems" here means applications built before GenAI went mainstream, roughly before 2023. Their observability rested on three kinds of data: logs, metrics and traces.

Logs and metrics are familiar. A trace is the end-to-end record of a single request or interaction, made of spans that describe the individual steps. In a microservices architecture, one trace might cover a whole application transaction, with spans for an API call, a database query or a call to an outside service. To be effective, the approach has to follow the request across every component it touches.

For GenAI agents, that classic model isn't enough by itself. Beyond logs, metrics and traces, you also have to observe what the model does, what context it received, which tools it called and how good its responses actually were.

What changes with GenAI

Conventional software is fairly predictable. Systems built on generative AI are less so, particularly when they have some freedom to make decisions. An LLM's output depends on far more than the user's input, and all those factors need to be visible.

An observability setup for an LLM system should capture at least four things:

• the model and the parameters of the call
• the instructions and the actual prompt sent
• the context provided, including earlier conversation history
• how the system executed, covering tool calls, intermediate outputs, latency and token usage

Looking only at the final answer falls short. To understand a GenAI system, you should be able to reconstruct, for any trace, everything that contributed to the response.

That's even more true once the model has operational freedom. If it can choose which tools to invoke, the order of actions, which sources to consult or when to stop, then prompts and outputs aren't enough. You need to see the decision process itself, including the sequence of tool calls, timing and intermediate results.

Even a complete trace often can't answer whether the output is any good. For that, each trace usually needs scores attached. They can come from conventional logic or from another LLM, the now common "LLM-as-a-judge" technique.

Consider judging a RAG answer. It can be partly right, or read as plausible without being backed by the retrieved sources. You then need to tell apart at least two issues: the root cause, such as retrieval, reranking or generation, and grounding, meaning whether the answer actually rests on the retrieved documents. Grounding checks reveal unsupported claims and therefore possible hallucinations.

All of this means a generic observability platform won't do for GenAI. You need a tool that captures this information natively and keeps the instrumentation effort low. The real difference is scope. For GenAI you watch infrastructure and speed as before, and in addition the context the model saw, the decisions it took, the quality of what it produced and how consistently it behaves.

From theory to a working setup

So observability for enterprise agents has to include prompts, context, tool calls, response times, token consumption, output quality and the likely causes of errors or poor answers, on top of the classical signals. The practical question is how to get all that in a simple, effective way that fits an enterprise setting. The answer below is an open-source one that's easy to adopt on OCI.

Langfuse

Among open-source tools for LLM and agent observability, Langfuse stands out. It's a platform for tracing, observability and evaluation (scoring) of LLM-based applications, and it covers much of the data discussed above: prompts, outputs, traces, metadata, response times, token usage, tool calls and evaluation scores. It's more than a logging service. It gives you a structured vantage point on how a GenAI application actually behaves, organized around traces and sessions rather than a flat stream of text lines that someone has to grep through.

Why OCI makes integration easy

OCI Enterprise AI Agents exposes an API that's fully compatible with the OpenAI Responses API. That matters beyond convenience, since it lets you reuse the tools, libraries and development patterns of the broad OpenAI-compatible ecosystem. For Langfuse, it sharply cuts the work of collecting telemetry from model calls and building the observability layer, so more time goes to analysis and less to plumbing. That's a big part of why OCI and Langfuse pair well in an enterprise.

A quick proof-of-concept deployment

For a PoC, Langfuse installs readily on OCI, for instance on a Linux virtual machine, using the official Docker Compose deployment from the Langfuse repository. That isn't necessarily the final architecture for a production-grade enterprise environment. It's more than adequate for validating the idea, gathering real telemetry and seeing what analysis becomes possible after integration. In practice it produced a working platform quickly, reachable from a browser and ready to receive traces from Python LLM applications.

Wiring up the code

With a Langfuse instance running, the next step is instrumenting the application so model-call telemetry flows in automatically. Because of the Responses API compatibility, this is simple for Python apps already written around an OpenAI-style call. The app keeps calling the model the way it always did, and Langfuse records the details that matter.

The changes are small:

1. Set a few environment variables.
2. Use the client supplied by the Langfuse library, which is already instrumented to collect and send traces.
3. Pass the conversation_id along as the session_id, so requests in one conversation are linked in the same session.
4. Optionally force a flush to send the trace right away. Otherwise it goes out automatically after a few seconds.

What gets collected

After integration, Langfuse collects a lot of useful information automatically. For agents, that includes more than model inputs and outputs. It also covers the execution flow, with intermediate spans, tool calls and timing that help you reconstruct how a result came about.

The most relevant items are:

• the model and its version
• the prompt sent
• the output generated
• metadata attached to the call
• the trace structure and any spans
• latency measures, including user-facing ones like time to first token
• input and output token counts, which feed cost analysis
• tool calls and their details
• scores and evaluations linked to the trace

Not everything useful arrives automatically, and that's by design, since the platform can't know what your business considers a good answer. Domain-specific scores, custom indicators and business metadata can, and sometimes must, be added by the application. The mix of automatic collection and deliberate enrichment is what makes the tool effective.

Sessions

Langfuse can link multiple traces from one user session through a session_id. In this setup the conversation is identified by the Responses API's standard conversation_id, so setting session_id equal to it correlates all requests from the same conversation. Sessions can then be browsed in the UI, where each one holds several traces.

Looking inside a trace

The payoff shows up during analysis. The Langfuse interface lets you explore each interaction in detail, surfacing things that would otherwise be scattered among application logs, loose metadata and code. For a single trace you can see the prompt actually sent, the model, its parameters, execution times, tokens consumed, the sequence of any calls and the outputs at each stage. That matters most when you're investigating odd behavior, latency problems, creeping costs or answers of unsatisfactory quality. A slow response, for example, might trace back to a long chain of tool calls, a bloated prompt or a slow first token, and each of those has a different fix. Without the span-level timing and token counts side by side, you'd be guessing which one it is.

A practical habit that falls out of this is to treat every conversation as something you can replay later. When a user reports a bad answer a week afterward, the session view lets you open the exact traces, read the real prompt and context and see which tool returned what, instead of trying to recreate the situation from memory.

Limits to keep in mind

Adopting Langfuse doesn't resolve every observability and evaluation challenge in a GenAI system. Rich context-laden traces are necessary but not always sufficient. The most useful scores usually depend on the application domain and have to be designed on purpose. And real enterprise environments force careful thought about security, sensitive data, telemetry retention, access control and deployment architecture.

Even so, an open-source platform plus the full API compatibility of OCI makes a strong starting point for a concrete observability solution for enterprise agents.

The bottom line

For enterprise AI agents, observability isn't a technical detail or an optional extra. It's a precondition for understanding behavior, analyzing errors, evaluating answer quality and holding response times and running costs down. Watching infrastructure, application logs and standard metrics doesn't go far enough. You also need data much closer to what the agent does: prompts, context, tool calls, token consumption, user-perceived latency and evaluation scores.

Langfuse is an especially interesting open-source option because it captures and organizes a large part of that telemetry, and the OpenAI-compatible API in OCI Enterprise AI Agents makes the integration easy. No platform solves everything alone, though. Security, governance, handling of sensitive data and the design of domain-specific scores all remain essential.

If these agents are going to move past demos and become dependable parts of real systems, observability can't remain an afterthought. Their complexity can't be removed, but it can and should be made visible.

A natural follow-up is enriching traces with scores that measure quality more systematically, particularly scores produced by an LLM-as-a-judge, so each trace carries evaluations of response quality alongside the technical execution data.