Tracing a LangGraph Blog Pipeline with Langfuse: Spans, Scores, Warnings

Guardrails (source checks, citation checks, capped retries, human escalation) only help if a failed run can be reconstructed afterward. A multi-agent blog pipeline invites plenty of questions at that point. Were the research sources weak? Did the writer ignore citation warnings? Was the reviewer right to reject? Did the search call time out? What did the run cost in model calls and tokens?

Without observability the system is a black box wrapped around further black boxes. Multi-agent tracing is a notoriously hard part of the job. This setup adds Langfuse to a LangGraph blog pipeline, producing one trace per article run that holds handoffs, model and tool calls, timings, costs, errors and evaluation results.

Deciding what a trace is

Before any code, pick the unit to inspect. In this pipeline, a trace equals one attempt at producing one article. It begins when a topic is submitted and ends when the pipeline finishes, pauses for a human or fails. Within it sit observations for each piece of work done:

blog-pipeline
├── research-agent
│   ├── search-web
│   └── research-model
├── writer-agent
│   └── writer-model
├── citation-check
│   └── citation-review-model
└── reviewer-agent
    └── reviewer-model

Langfuse works with three nouns. A trace is one full run from start to finish. An observation is a single step inside it, which may be a span (application logic), a generation (an LLM call) or an event (a tiny marker). A session ties multiple traces together, like all attempts on the same blog topic.

The hierarchy mirrors how debugging goes: start from the full run, then drill into the specific agent, tool or model call that caused trouble.

Setting up Langfuse

Create a Langfuse project and supply its credentials through environment variables. The secret key should never be written into source files.

pip install -U langfuse

export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_BASE_URL="https://cloud.langfuse.com"
export LANGFUSE_TRACING_ENVIRONMENT="development"

The langfuse package is the Python SDK. The two keys authenticate the app against the project, and LANGFUSE_BASE_URL picks the deployment. The sample targets the EU cloud region, so swap it for another region or a self-hosted instance as needed.

An environment label keeps development traces apart from production ones. Short, stable names such as development, staging and production work well. Without them, experiments and real article runs land in one view and debugging turns confusing.

Hooking in the callback handler

Langfuse ships a LangChain callback handler. LangGraph builds on LangChain's runnable and callback machinery, so the same handler captures graph, chain, tool and model events:

from langfuse import get_client, propagate_attributes
from langfuse.langchain import CallbackHandler


langfuse = get_client()
langfuse_handler = CallbackHandler()

get_client() builds a client from the environment variables. The handler should be created once at startup and reused for each run. It listens to LangChain events and forwards them as observations, which removes any need to trace each model call by hand.

propagate_attributes comes into play next. It stamps the same labels on every observation within one run, which keeps a search call, a writer generation and a reviewer generation filterable as parts of one article pipeline.

The graph invocation is wrapped in a root span, with the handler passed through the LangGraph config:

def run_blog_pipeline(topic: str, blog_id: str, user_id: str):
    initial_state = {
        "topic": topic,
        "audience": "developers new to agent systems",
        "research_brief": "",
        "sources": [],
        "open_questions": [],
        "article_draft": "",
        "review_feedback": "",
        "citation_issues": [],
        "approved": False,
        "revision_count": 0,
        "status": "researching",
    }

    with langfuse.start_as_current_observation(
        as_type="span",
        name="blog-pipeline",
        input={"topic": topic, "audience": initial_state["audience"]},
    ) as pipeline_span:
        with propagate_attributes(
            trace_name="blog-pipeline",
            session_id=blog_id,
            user_id=user_id,
            tags=["blog-pipeline", "langgraph"],
            version="1.0.0",
            metadata={"workflow": "research-write-review"},
        ):
            trace_id = langfuse.get_current_trace_id()
            result = graph.invoke(
                initial_state,
                config={"callbacks": [langfuse_handler]},
            )

        pipeline_span.update(
            output={
                "status": result["status"],
                "approved": result["approved"],
                "revision_count": result["revision_count"],
            }
        )

    return result, trace_id

Several details of that wrapper matter. The outer start_as_current_observation() call creates the root observation, and its input stays deliberately compact, just the topic and audience. That's what shows in the trace list at a glance, so a dump of internal state doesn't belong there.

propagate_attributes() tags every nested observation. The blog_id serves as the session ID, grouping retries or repeated attempts on one article. A version value identifies the workflow and prompt release behind a result, so when behavior shifts, traces can be filtered by version and not by guesswork.

The handler travels through LangGraph's regular config, and from there it collects framework-level detail automatically: each node call, LLM generation, tool call, input, output, latency, token count and cost.

At the end, the root span is updated with three pipeline-level outcomes: whether the run completed, whether the draft was approved and how many revisions were consumed. The returned trace_id should be stored in application logs or the UI, giving a direct jump from an article run to its Langfuse trace.

Business-level spans inside agents

Automatic tracing proves that a model was called. It doesn't show what each specialist passed to the next. For that, add a light span inside an agent node, and forward the config argument into nested LangChain or LangGraph calls so the handler keeps the same trace context:

from langchain_core.runnables import RunnableConfig


def research_node(state: BlogState, config: RunnableConfig) -> dict:
    with langfuse.start_as_current_observation(
        as_type="span",
        name="research-agent",
        input={"topic": state["topic"], "audience": state["audience"]},
    ) as span:
        result = research_agent.invoke(
            {"topic": state["topic"], "audience": state["audience"]},
            config=config,
        )

        span.update(
            output={
                "source_count": len(result["sources"]),
                "open_question_count": len(result["open_questions"]),
                "brief": result["research_brief"],
            }
        )

    return {
        "research_brief": result["research_brief"],
        "sources": result["sources"],
        "open_questions": result["open_questions"],
        "status": "writing",
    }

The span complements the callback handler and doesn't replace it. It adds the meaning the framework can't guess, namely what "research succeeded" means for this pipeline. Here it records the source count, the number of unresolved questions and the brief handed to the writer.

Passing config: RunnableConfig into research_agent.invoke() carries callbacks and trace context along. Forgetting it is a classic observability bug: the top-level graph is traced, yet the nested call shows up as an unrelated trace or vanishes.

Keep observation names constant: research-agent, writer-agent, citation-check. A topic, model name or user ID shouldn't be baked into a name, because that data fits input, model attributes or metadata. Constant names mean dashboards and filters survive implementation changes.

Warnings and errors

Not every odd outcome is a crash. A dubious source, a spent revision budget or an unsupported citation matters too, and should surface as a warning.

def record_source_assessment(assessment: SourceAssessment) -> None:
    if assessment.suspicious_content:
        langfuse.update_current_span(
            level="WARNING",
            status_message="Untrusted source contained agent-directed instructions.",
        )


def record_pipeline_failure(error: Exception) -> None:
    langfuse.update_current_span(
        level="ERROR",
        status_message=f"Pipeline failed: {type(error).__name__}",
    )

Langfuse supports log levels on observations. WARNING fits a run that finished but needs a look, while ERROR is for real failures. Status messages should describe the operational issue without copying sensitive source text or secrets into telemetry.

The source-assessment recorder belongs in the source-checking path, and the failure recorder in an error handler that can actually recover or report. Catching every exception just to log it is a bad idea, since unexpected errors should still reach error monitoring and fail loudly.

Scoring the outcome

A trace says what happened, and a score says whether the result was any good. Once a run completes, attach a first deterministic score derived from pipeline state:

result, trace_id = run_blog_pipeline(
    topic="How AI agents use tools",
    blog_id="blog-ai-tools-001",
    user_id="philip",
)

if trace_id:
    langfuse.create_score(
        trace_id=trace_id,
        name="review_approved",
        value=1 if result["approved"] else 0,
        data_type="BOOLEAN",
        comment=result["status"],
    )

    langfuse.create_score(
        trace_id=trace_id,
        name="revision_count",
        value=float(result["revision_count"]),
        data_type="NUMERIC",
    )

The first score captures the reviewer's structured approve-or-reject verdict, and the second measures how much rework a draft required. Accumulated over time, they reveal more than anecdotes. Which model, prompt version, topic category or sourcing approach yields fewer revisions and more approvals?

Neither is ground truth, since a reviewer can be wrong. A human score added after reading the final draft can come later. The habit that counts is making quality measurable and comparing across traces, rather than trusting a handful of memorable examples.

A debugging routine

When an article turns out weak, open its trace and ask these questions in order:

1. Did the graph take the expected route? Look for Research, Writer, Citation Check and Reviewer, plus any revision loop.
2. How solid was the research handoff? Review the number of sources, open questions, citations and the brief itself.
3. Where did the defect enter? Set each call's prompt, output, latency, token use and tool activity against the step before it.
4. Model fault or system fault? A timeout and a flimsy citation need entirely different remedies.
5. Did the guardrail hold? If a source looked suspicious or a claim lacked support, did the graph route to revision or human review rather than approval?

That's why trace shape matters. A well-named tree converts "the blog was wrong" into a precise finding, such as "the writer added a claim missing from the brief after the second revision."

Keeping sensitive data out

Tracing can record prompts, outputs, tool results and metadata, which also means it can capture information that shouldn't be stored. API keys, authentication headers and full secret configuration should never go to Langfuse. If user data enters the pipeline later, review what gets captured and set up masking before exporting traces. Langfuse offers masking functions that redact sensitive values before they leave the application.

The rule of thumb: record enough to explain a decision, and no more than can be safely retained.

What the pipeline gains

Each run now leaves an operational record:

• Every article attempt is a single trace.
• Agents, models and tools show up as nested observations.
• Constant names plus metadata, version, session and environment make filtering easy.
• Questionable sources trigger warnings, and true failures are flagged as errors.
• Approval and revision count are recorded as scores.
• A trace ID leads directly to the run behind any given article.

Debugging stops being a matter of scrolling terminal logs and guessing, and becomes an inspection of the exact handoff where things broke.