Agentic RAG: Letting the LLM Decide When and Where to Retrieve

"RAG is mostly a solved problem." That line gets said at team reviews, and the honest reply is: it depends on what you mean by retrieval. Anyone who has run RAG pipelines in production for a while, on healthcare record summaries, natural-language-to-SQL workflows or multi-agent research systems, tends to hit the same ceiling.

The ceiling isn't the embeddings. It's the assumption that retrieval happens once, at a fixed step, against one source.

Static RAG (embed your documents, fetch the top-k, hand them to the LLM) handles simple, self-contained questions over stable internal knowledge well. Once a question turns multi-hop, time-sensitive, or needs both internal documents and live data, the static pipeline starts producing confident hallucinations. A pipeline like that can recommend a clinical procedure off stale internal guidance, even though a quick half-minute web search would have exposed the change.

Better embeddings won't fix that. What helps is letting the LLM decide when and where to retrieve.

The trouble with "retrieve once, answer always"

Imagine a librarian who can only answer by walking to a single shelf, the same one every time, before you've finished your sentence.

Ask "What are the side effects of metformin?" and the shelf delivers. Now ask something harder: which drug interactions has the FDA recently flagged for metformin alongside newer GLP-1 medications, and does the current patient protocol cover them? That needs internal documents and live regulatory data, plus reasoning about the gap between them. Static RAG pulls once from one source, and the LLM glosses over whatever it received.

This isn't hypothetical. In an internal evaluation of a healthcare pipeline [1], static RAG reached 89% answer relevance on stable questions but fell to 61% when a question required cross-referencing internal clinical documents with external treatment guidelines. Basic evals missed it because the answers sounded right. They were just stale.

The fix is easy to state and fiddly to build: retrieval becomes a decision rather than a step.

What agentic RAG means

In agentic RAG, the LLM doesn't retrieve for you at a preset point. It treats retrieval as a tool, like a calculator or code executor, and calls it when it judges the evidence insufficient. Three behaviors follow that a static pipeline can't produce.

• Conditional retrieval. If the context already answers the question, the agent skips retrieval altogether.
• Multi-hop retrieval. The agent fetches something, reads it, realizes a piece is missing and goes back for more, with each pass shaped by what the previous one found. That's how people research.
• Source routing. The agent picks between a vector store, web search or a structured database depending on the question. "What does our policy say?" goes to the vector store, while "What did the FDA announce last week?" goes to the web.

None of this is new as an idea. The ReAct paper by Yao and colleagues (2022) set out the pattern of interleaving reasoning steps with tool calls, and agentic RAG is ReAct applied to retrieval [2].

The recent change is in the tooling. LangGraph finally makes stateful loops with branching practical. Tavily offers a web search API designed for agents that returns LLM-ready content, and it reports handling over 100 million requests a month with a p50 search latency of 180 ms [3]. Details are in the Tavily 101 post.

Three moving parts

Think of a detective instead of a librarian. A librarian fetches on command. A detective reads what's on hand, works out what's missing, goes to get it, reads again and continues until a conclusion can be defended. The loop has three components.

1. The router answers what kind of retrieval a query needs. The LLM looks at the question and decides whether existing context suffices, whether documents are needed, whether the web is needed, or both.
2. The retrieval tools are the evidence sources. Two tools are the bare minimum: one searching a vector store of internal documents, one searching the web for live information. Each is an ordinary function that the LLM invokes with a query string.
3. The grounding check asks whether the evidence is enough. Before writing the final answer, the agent assesses whether what it retrieved is sufficient and relevant, and if not, it retrieves again. Static pipelines can't express this loop.

A working implementation

Here's a hybrid agent in LangGraph that switches between a Qdrant vector store and Tavily web search. It chooses the tool, can call both in sequence and will fetch again when the first result doesn't ground the answer. The Qdrant side follows the LangChain integration guide.

from typing import Annotated, Literal
from langgraph.graph import StateGraph, START, END
from langgraph.prebuilt import ToolNode
from langchain_core.messages import HumanMessage, AIMessage, SystemMessage
from langchain_core.tools import tool
from langchain_openai import ChatOpenAI
from langchain_qdrant import QdrantVectorStore
from langchain_openai import OpenAIEmbeddings
from tavily import TavilyClient
from qdrant_client import QdrantClient
import operator
from typing import TypedDict, Sequence

# --- State definition ---
class AgentState(TypedDict):
    messages: Annotated[Sequence, operator.add]
    retrieval_count: int  # guard against infinite loops

# --- Tool 1: Vector store retrieval (internal docs) ---
qdrant_client = QdrantClient(url="http://localhost:6333")
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vector_store = QdrantVectorStore(
    client=qdrant_client,
    collection_name="internal_docs",
    embedding=embeddings,
)
@tool
def retrieve_from_docs(query: str) -> str:
    """
    Search internal document knowledge base.
    Use for questions about internal policies, protocols, or proprietary knowledge.
    """
    results = vector_store.similarity_search(query, k=4)
    if not results:
        return "No relevant internal documents found for this query."
    return "\n\n---\n\n".join([
        f"[Source: {doc.metadata.get('source', 'internal')}]\n{doc.page_content}"
        for doc in results
    ])

# --- Tool 2: Web retrieval (live data) ---
tavily = TavilyClient(api_key="YOUR_TAVILY_API_KEY")
@tool
def retrieve_from_web(query: str) -> str:
    """
    Search the live web for current information.
    Use for recent news, regulatory updates, or anything that may have changed
    since the internal knowledge base was last updated.
    """
    results = tavily.search(
        query=query,
        max_results=5,
        search_depth="advanced",  # deep retrieval for better accuracy
    )
    if not results.get("results"):
        return "No relevant web results found."
    return "\n\n---\n\n".join([
        f"[Source: {r['url']}]\n{r['content']}"
        for r in results["results"]
    ])

# --- Agent node ---
tools = [retrieve_from_docs, retrieve_from_web]
tool_node = ToolNode(tools)
llm = ChatOpenAI(model="gpt-4o", temperature=0).bind_tools(tools)
SYSTEM_PROMPT = """You are a research assistant with access to two retrieval tools:
- retrieve_from_docs: internal document knowledge base (policies, protocols, proprietary data)
- retrieve_from_web: live web search (recent news, regulatory updates, real-time information)
For any question:
1. Decide whether you need to retrieve information or can answer from context.
2. If retrieval is needed, choose the right source - or call both if the question spans internal and live knowledge.
3. After retrieving, check if the evidence is sufficient. If not, retrieve again with a refined query.
4. Always cite your sources in the final answer.
Do not hallucinate. If you cannot find sufficient evidence, say so explicitly."""
def call_agent(state: AgentState) -> AgentState:
    messages = [SystemMessage(content=SYSTEM_PROMPT)] + list(state["messages"])
    response = llm.invoke(messages)
    return {
        "messages": [response],
        "retrieval_count": state.get("retrieval_count", 0)
    }
def update_retrieval_count(state: AgentState) -> AgentState:
    return {
        "messages": [],
        "retrieval_count": state.get("retrieval_count", 0) + 1
    }

# --- Routing logic ---
MAX_RETRIEVALS = 3  # prevent infinite loops
def should_continue(state: AgentState) -> Literal["tools", "end"]:
    last_message = state["messages"][-1]
    # If the LLM called a tool and we haven't hit the loop guard, continue
    if hasattr(last_message, "tool_calls") and last_message.tool_calls:
        if state.get("retrieval_count", 0) < MAX_RETRIEVALS:
            return "tools"
    return "end"

# --- Build the graph ---
graph_builder = StateGraph(AgentState)
graph_builder.add_node("agent", call_agent)
graph_builder.add_node("tools", tool_node)
graph_builder.add_node("count_retrieval", update_retrieval_count)
graph_builder.add_edge(START, "agent")
graph_builder.add_conditional_edges(
    "agent",
    should_continue,
    {"tools": "tools", "end": END}
)
graph_builder.add_edge("tools", "count_retrieval")
graph_builder.add_edge("count_retrieval", "agent")
agent = graph_builder.compile()

# --- Run it ---
if __name__ == "__main__":
    result = agent.invoke({
        "messages": [
            HumanMessage(content=(
                "What does our current patient protocol say about GLP-1 usage, "
                "and are there any FDA safety updates issued in 2025 we should be aware of?"
            ))
        ],
        "retrieval_count": 0,
    })
    print(result["messages"][-1].content)

Some details deserve attention.

The MAX_RETRIEVALS guard in the routing logic is no decoration. Without it, agents can spiral into 15 retrieval calls over one ambiguous question, wasting tokens and patience. Three retrievals usually suffice. If they don't, the question is too broad or the tools are returning junk.

The retrieval_count field in state is what makes the loop breakable. LangGraph's StateGraph carries it across every node transition, so it persists through the whole conversation turn and not just a single cycle.

The system prompt carries real weight too. Telling the model when to use each tool, not merely that tools exist, separates an agent that routes sensibly from one that calls retrieve_from_web for everything out of laziness.

When agentic RAG makes things worse

It doesn't replace static RAG. It replaces it for particular problem shapes, and it backfires in four cases.

• Latency is the main constraint. Each pass through the loop costs one more model call plus a tool call. For a basic Q&A assistant where p95 latency has to stay under a second, an agent that may cycle two or three times is a non-starter. A well-tuned static retriever is faster and enough.
• The knowledge base is stable and self-contained. If everything users ask can be answered confidently from the vector store, you're adding complexity for nothing. Measure the static pipeline with RAGAS first (paper). If context recall is above 85% and answer faithfulness above 90%, the agentic wrapper is overhead.
• You can't trust the model's tool choices. With a carefully written system prompt, GPT-4o and Claude Sonnet route correctly between document and web retrieval roughly 88% of the time, by the same internal measurements. The other 12% creates source errors that are harder to debug than plain hallucinations, because the answer is grounded, just in the wrong place. With a weaker model, instrument tool selection before shipping.
• Real-time data can't be trusted. Web retrieval adds a live attack surface. Tavily filters for prompt injection inside its API, but no filter is perfect, so anything an agent does based on web content should treat that content as potentially hostile.

Reach for agentic RAG when knowledge spans several sources and some of them change. Start with static RAG and upgrade when RAGAS scores expose a specific failure this design solves.

Where to begin

Smallest step, under five minutes: add one Tavily tool to an existing LangChain agent with from langchain_community.tools.tavily_search import TavilySearchResults. You don't need LangGraph yet. Expose the tool, write a system prompt saying when to call it, run five test queries and watch what it picks.

Next level, about an afternoon: point the LangGraph example above at your own vector store, keeping both tools. Leave out the retrieval counter at first to see how often the agent loops, then add it. The difference teaches a lot.

Advanced: insert a grading step after each tool result and before the agent's next turn. A smaller, cheaper LLM call answers yes or no to the question of whether the retrieved context is enough, and only then does the main model compose its reply. That step moves context recall from "good enough in testing" to "reliable in production."

Tool-routing errors, that 12% above, are where most production debugging time goes. Instrument them early, and let the grading node catch them before users do.

Sources

[1] Internal eval runs on a healthcare RAG pipeline using RAGAS 0.1.x against a curated set of 200 clinical queries. Not a published study.

[2] Yao et al. (2022), ReAct: Synergizing Reasoning and Acting in Language Models, arXiv:2210.03629.

[3] Tavily blog, Tavily 101, with the latency and volume figures as stated on the product page in May 2026.