Tracing the Retrieval Step: Six Observability Tools Compared on One RAG Agent

Retrieval-augmented generation is a two-step pipeline, and only one of the steps is a model call. To see how observability tools record the other half, the same small agent was instrumented through six platforms: Opik, MLflow, Langfuse, Phoenix, AcruxCore and Helicone.

Why retrieval slips past the usual wrapper

A single model call is simple to observe: patch the OpenAI client and each chat.completions.create() appears as a span. Adding retrieval seems like a minor step. Context gets fetched, the model gets called, and the same patch should still cover everything. In practice it covers only half.

Put differently, a RAG pipeline is at least two separate things to trace. One is the retrieval step, which no client wrapper notices. The other is the generation step, which resembles any ordinary model call. The experiment aimed straight at that seam, holding everything else fixed between runs.

The findings in brief

• Decorators dominate. Opik, MLflow and Langfuse each turn a function into a traced step with one added line, leaving the body untouched.
• OpenTelemetry works in two places. Phoenix and AcruxCore both take OTLP, so one handwritten span can be sent to either by switching the endpoint. The same pipeline was run against both.
• Two products live in the request path. Helicone's proxy and AcruxCore's gateway record the model span with no client code, but neither notices a step that never contacts a model provider.
• Helicone has no answer for retrieval, since the proxy is its whole product and no SDK exists to log that step.
• MLflow offers the richest span vocabulary, fifteen types among them RERANKER, GUARDRAIL and EVALUATOR, counted from the SDK source rather than the documentation.

What a span type is

Think of a trace as the list of stages a request passed through. Every stage is a span carrying a name, start and end times, and its input and output. That reveals what ran and how long it lasted, yet not the nature of the work: a span labeled step_2 lasting 900 ms could equally be a SQL lookup, an LLM invocation or an HTTP fetch. Telling them apart is the job of the span type, a tag that settles it. Each tool defines a closed set of categories, and every span gets one.

Take the RAG agent here. One request yields three spans:

┌────────────────┬────────────────────────────────┬────────────────┐
│   Span name    │          What it did           │   Span type    │
├────────────────┼────────────────────────────────┼────────────────┤
│ rag-agent      │ the whole request              │ agent or chain │
├────────────────┼────────────────────────────────┼────────────────┤
│ search_docs    │ queried Chroma for four chunks │ retrieval      │
├────────────────┼────────────────────────────────┼────────────────┤
│ ChatCompletion │ called gpt-4o-mini             │ llm            │
└────────────────┴────────────────────────────────┴────────────────┘

Here search_docs is simply a name the developer picked, while retrieval comes from the list of values the SDK allows. A viewer shows this as a Retrieval chip ahead of search_docs and an LLM chip ahead of the completion.

The type matters, and not only the name, because the tool acts on it. It chooses the icon and layout, so a retrieval span shows the fetched documents while a model span shows token count and cost. It enables filtering, so a question like which retrieval steps took over a second works across all traces even when developers named their functions differently. And it drives analytics: how much latency comes from retrieval versus generation becomes a query on the type column.

The vocabularies vary in length. AcruxCore allows seven: llm, tool, retrieval, embedding, agent, chain and other. MLflow allows fifteen, among them RERANKER, GUARDRAIL, EVALUATOR, PARSER and MEMORY. Opik allows only four: general, tool, llm and guardrail.

The agent under test

The agent is intentionally dull. It indexes five pages of OpenTelemetry documentation, cut into roughly 1,000-character pieces, embeds them with text-embedding-3-small and stores them in an in-memory Chroma collection. A single fixed question about what a span is and how parent and child spans make up a trace pulls back four chunks, and gpt-4o-mini writes the answer from those.

None of the six platforms owns the documents, on purpose. If the agent answered questions about one of the tools being compared, the test would be circular, and the answer text appears in every screenshot.

def retrieve_context(collection, question: str, top_k: int = 4) -> str:
    [query_embedding] = embed_texts([question])
    results = collection.query(query_embeddings=[query_embedding], n_results=top_k)
    chunks = results["documents"][0]
    sources = [m["source"] for m in results["metadatas"][0]]
    return "\n\n".join(f"[{src}]\n{chunk}" for src, chunk in zip(sources, chunks))

This function, the chunking step and the embedding call do not change across the six runs. The only variable is the instrumentation wrapped around them. Every tool ran on a local machine: five self-hosted in Docker, plus AcruxCore. LangSmith and W&B Weave were skipped because they exist only as hosted services. Judging by its SDK, LangSmith would be a fourth decorator example with seven run types, though with no trace actually observed there is nothing further to claim about it.

Pattern 1: the decorator

This is where the tools converge, and sensibly so: the function body stays exactly as written. Code that already exists just gets annotated. Opik's form matches its docs:

@opik.track(name="search_docs", project_name=PROJECT_NAME)
def search_docs(collection, query: str) -> str:
    return rag_core.retrieve_context(collection, query)

MLflow has the same shape with a span type attached:

@mlflow.trace(name="search_docs", span_type=SpanType.RETRIEVER)
def search_docs(collection, query: str) -> str:
    return rag_core.retrieve_context(collection, query)

Langfuse provides @observe, easy to overlook when its context-manager examples are encountered first:

@observe(name="search_docs", as_type="retriever")
def search_docs(collection, query: str) -> str:
    return rag_core.retrieve_context(collection, query)

The model call nests automatically in all three. Each supplies a patched OpenAI client (track_openai(), mlflow.openai.autolog() and langfuse.openai), so the generation span hangs off whichever decorated function is currently running. A single decorator therefore gives a tree two levels deep.

Langfuse produced the tidiest output. Its retriever-typed search_docs span lasted 1.05 s, and the generation span it captured on its own lasted 4.14 s, with cost and token totals rolled up. In MLflow the same two stages show retrieval at 0.47 s and the captured Completions span at 3.37 s. Opik draws a correct tree as well, apart from a cosmetic oddity: trace and root span share one name, so rag-agent is listed twice above search_docs.

Pattern 2: a vendor context manager

Langfuse also exposes the span object itself, which is the tool to reach for when a span's boundaries do not match a function's boundaries:

with langfuse.start_as_current_observation(name="search_docs", as_type="span") as span:
    context = retrieve_context(collection, QUESTION)
    span.update(input={"query": QUESTION}, output={"context": context})

What is gained is precision, since input and output can be set at the exact right instant and a span can cover part of a function. What is lost is separation, because tracing code ends up inside business logic. When a whole function is the unit, @observe achieves the same result without that downside.

Pattern 3: standard OpenTelemetry

Phoenix has no decorator of its own. The OpenTelemetry tracer is used directly, and the OpenInference instrumentor wraps the client automatically so the generation call still nests:

with tracer.start_as_current_span("search_docs") as span:
    context = retrieve_context(collection, QUESTION)
    span.set_attribute("output.value", context)

The benefit is that nothing here belongs to Phoenix. These are standard OTel spans, so the same instrumentation can feed any OTLP-compatible backend once the endpoint is changed. That claim got a real check, since portability is often true on paper and broken in practice. AcruxCore speaks OTLP too, so the very same pipeline was aimed at it: identical with tracer.start_as_current_span(...) body, identical instrumentor, new endpoint.

tracer_provider = otel.register(service_name="rag-agent-observability")
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

Each platform has a register() helper that folds a TracerProvider, a BatchSpanProcessor and an OTLP exporter into one call. Neither helper is mandatory, and hand-built pipelines work with both. The spans came through correctly nested. The OpenInference RETRIEVER attribute became a Retrieval kind, and the automatically captured model span listed the model and 1,341 tokens.

So portability holds up, and it is the best reason to write instrumentation in plain OTel: the spans get written once, and which backend receives them turns into a settings question. The cost sits up front. A register() call, an instrumentor and a tracer object all precede the first span, which is more scaffolding than any decorator needs for the same two spans.

Pattern 4: out-of-band ingest

Besides OTLP, AcruxCore offers a fifth route that the other tools lack. Instead of watching the process, the code describes the spans and posts them. The trace id ties things together: pass it to the gateway, and the model span written server-side joins the same trace. The two routes complement each other.

reported = await hub.traces.ingest({
    "name": "rag-agent",
    "spans": [{
        "spanId": "search_docs", "name": "search_docs", "kind": "retrieval",
        "input": {"query": QUESTION}, "output": {"context": context},
        "startTime": started, "endTime": _now(),
    }],
}, wait=False)                                # buffered, returns at once
result = await hub.gateway.chat(
    "openai/gpt-4o-mini", messages, provider=PROVIDER,
    trace={"trace_id": reported.trace_id},   # generation lands in the same trace
)

It needs fewer calls than building a tracer, and timestamps are set by the caller, which fits work that has already finished and left no live call stack to attach to.

The catch is the wait=False flag. In the default mode ingest() waits for its POST and hands back a trace id minted by the server, so the request pays a network round trip before the model call begins. With wait=False the span lands in a local SDK buffer and the id is created client-side. Nothing is acknowledged at the call site, and a send that fails is logged as a warning and discarded, much like an OTel exporter. The decision is between slow and loud on failure or fast and silent on failure. Either is defensible, as long as the choice is deliberate.

Pattern 5: sitting in the request path

Two of the tools live inside the call itself, and the model span comes free as a consequence. Helicone shows this most plainly: change base_url to point at it and the completion is recorded with zero SDK code. The AcruxCore gateway takes the same stance. A call routed through gateway.chat() has its llm span written by the server, so the client needs no instrumentation.

The position has one benefit and one limit, shared by both. Anything in the request path sees only HTTP traffic bound for a model provider. A Chroma query sends no such traffic, so it stays invisible whichever tool is used.

They differ in what accompanies them. The AcruxCore gateway takes a trace id, which lets a span recorded through another route, OTel or ingest(), serve as parent so that the server-written model span lands in the same trace. Helicone offers only the proxy and no SDK to wrap an arbitrary function. That leaves two options, omitting retrieval from the trace or inventing a fake completion to carry it, and the latter fills telemetry with fiction. The accurate claim is therefore narrower than proxies being unable to trace retrieval: a proxy on its own cannot, and the importance of that depends on whether a second route exists.

One deployment caveat, unrelated to design: the self-hosted Helicone build in this test answered its manual-log endpoint with a 500 because an S3_REGION variable was absent, which blocked recording even the model half. That is a flaw of this particular image and not of the approach.

What can be called a span

Two spans show that the plumbing works, but they cannot distinguish a reranker from a guardrail half a year later. The allowed values were therefore pulled from each installed SDK and not from its docs.

MLflow takes this category by a wide margin. RERANKER, GUARDRAIL, EVALUATOR, PARSER and MEMORY are all built in, while Opik has four types and pushes anything else into metadata. AcruxCore's seven handle RAG but nothing more granular. Phoenix inherits its list from the OpenInference convention, so it is common to every tool that follows that standard and is not a Phoenix feature.

This is the row most likely to be misjudged from marketing pages. Any of the lists can be checked against installed versions in a few lines:

from mlflow.entities import SpanType
from openinference.semconv.trace import OpenInferenceSpanKindValues

print([v for k, v in vars(SpanType).items() if isinstance(v, str) and not k.startswith("_")])
print([e.value for e in OpenInferenceSpanKindValues])

Langfuse takes extra care. Its ObservationTypeLiteral, found in langfuse._client.span, is assembled from nested Literal types, so typing.get_args yields inner aliases instead of strings and the result must be flattened recursively. This explains why counts from its documentation often disagree with the SDK.

Choosing

• MLflow suits teams wanting precise semantics now and evaluation later. Its vocabulary is far the widest, though the stack is heavier than a pure tracing tool.
• Opik gets a correct tree running the same afternoon, with the least ceremony and just four span types.
• Langfuse gives both a decorator and a manual escape hatch with strong typing, at the cost of two sets of documentation.
• Phoenix fits teams that might change backends or already run OTel. It involves no vendor API and needs the most setup before a first span appears.
• AcruxCore fits when the model span should be written for you while retrieval still needs a record. It has three composable routes, a vocabulary smaller than MLflow, Phoenix or Langfuse, and a buffered ingest that fails silently.
• Helicone fits a pipeline that truly is only a model call. Capture takes no code, and a non-model step has nowhere to be recorded.

No ranking is the real lesson here. Asking whether a tool traces retrieval turned out to contain five separate questions, and every answer is a design decision that carries a visible price: a line of code, a with block, setup work, a network hop or a blind spot. Choose the price that hurts least, and inspect the span vocabulary before committing, because that is where surprises show up late.