Why an LLM Call Is Not an App: Where LangChain Fits in a RAG Pipeline

Calling a large language model is easy: send a prompt, get text back. Building something useful around that call is not. A real application has to ingest documents, find the passages that matter, keep track of the conversation, assemble prompts and present all of it through an interface, and the model is only one of those moving parts. This article explains what LangChain is by walking through exactly that surrounding machinery, using a document-reading assistant as the running example. Afterwards you should be able to describe each stage of a retrieval pipeline, say which parts a framework like LangChain takes off your hands, and judge whether it or one of its alternatives suits your project.

What LangChain is, in one paragraph

LangChain is an open-source framework for building applications on top of LLMs. Rather than giving you a model, it gives you modular building blocks and end-to-end tooling for the pieces that surround a model: prompt templates, output parsers, document loaders, retrievers, memory, tools and the glue that connects them. It works with the major model providers, integrates with a large catalogue of third-party tools, is free to use and is under active development. Typical things teams build with it include chatbots, question-answering systems, retrieval-augmented generation (RAG) and autonomous agents.

The key mental shift is that LangChain is not the LLM. It is the layer that lets the LLM work together with your data, your prompts and your users. If you only ever send one prompt and print the reply, you do not need it. The moment your application has several steps, you start writing the kind of plumbing it already provides.

A map of the territory

It helps to see the whole landscape before zooming in. Learning LangChain usually breaks down into three areas, each building on the previous one.

Fundamentals

These are the pieces every LangChain application touches:

• the overall component model
• models, meaning the wrappers around chat and completion APIs
• prompts and prompt templates
• output parsing, so free-form text becomes structured data
• Runnables and the LangChain Expression Language (LCEL), the composition layer
• chains, which connect steps into a workflow
• memory, for carrying context between turns

Retrieval-augmented generation

RAG is how you let a model answer from your own documents. The relevant components are:

• document loaders
• text splitters
• embeddings
• vector stores
• retrievers
• assembling all of the above into a working RAG application

Agents

Agents let the model decide which actions to take. The topics here are:

• tools and toolkits
• tool calling
• building an agent that uses them

The rest of this article concentrates on the first two areas, since they explain why the framework exists in the first place.

The problem LangChain solves

A production LLM system rarely consists of a single request. Look at what even a modest assistant has to handle:

• ingesting documents
• running semantic search
• generating and storing embeddings
• performing retrieval-augmented generation
• managing context and conversation state
• orchestrating one or more LLM calls
• serving a chat interface

Each of these is manageable alone. Wired together by hand, they turn into a tangle of custom code where changing the model, the vector store or the prompt format means touching several files. LangChain's value is in giving each concern a standard interface and a set of reusable abstractions, so the pieces snap together into a pipeline and can be swapped independently.

A worked example: an AI book reader

Consider an application where users upload books or PDFs, read them in a built-in reader, and ask an assistant questions about what they are reading. Picture a machine-learning textbook: a student might ask about the bias-variance tradeoff, how a particular CNN architecture is structured, what backpropagation does, how attention mechanisms work or which optimisation algorithm suits a problem.

The assistant can only answer well if it sees the right pages. That single requirement pulls in a whole chain of work: load the uploaded file, find the passages relevant to the question, keep the chat history coherent, build a prompt that combines the question with those passages, and send it to the model.

In this application LangChain would take care of:

• loading and parsing the uploaded documents
• connecting the chat interface to the LLM
• managing prompt templates
• keeping conversational context between questions
• building the retrieval pipeline that selects relevant text

This is the clearest illustration of why an LLM alone does not make an application. The model supplies language ability; everything that makes the answer about this particular book comes from the components around it.

Semantic search: finding text by meaning

The heart of the book reader is the ability to pull the right passages from a large collection. Keyword search struggles here, because a student's question rarely uses the same words as the textbook. Semantic search solves this with embeddings: numeric vectors that place pieces of text with similar meaning close together in a high-dimensional space. Searching then means finding the stored vectors nearest to the vector of the question.

A simple example

Suppose the query is "What is the capital of France?" A semantic search system does not look for documents that merely share tokens with the question. It looks for the passage whose meaning is closest, which is a paragraph about Paris, rather than passages about Berlin or Madrid, even though those also discuss European capitals and might score well on shared words.

That is the practical difference. A keyword engine ranks by overlapping terms; a semantic engine ranks by how close the meanings are. In practice many production systems combine the two, because exact terms such as product codes or error messages still benefit from keyword matching.

Why it matters for LLM applications

Models answer far better when given relevant context. Good retrieval therefore leads directly to:

• better document retrieval
• more accurate answers
• smarter recommendations
• assistants that respond with awareness of the user's material

LangChain does not implement vector search itself. It integrates with the parts you need for it: embedding models, vector databases, retrievers and similarity search, all exposed through consistent interfaces.

From question to grounded answer in six steps

Once documents are searchable, answering a question follows a predictable sequence:

• The user asks. A question arrives in natural language.
• The question is embedded. It is converted into a vector so it can be compared by meaning rather than by exact words.
• Relevant text is retrieved. The system fetches the chunks or pages whose vectors are closest.
• The input is assembled. The retrieved passages and the original question are combined into the prompt for the model.
• The model processes it. The complete prompt, context plus question, goes to the LLM.
• A grounded answer comes back. Because the model is reasoning over supplied text instead of memory alone, the response is more accurate and easier to trace to its source.

Every arrow in that list is a handoff between components, and that is exactly what LangChain is designed to manage. It simplifies the retrieval pipeline, chains the prompt steps together, handles memory, injects context into prompts and orchestrates the calls to the model. This six-step flow is the core of any RAG system. If you want a deeper look at retrieval itself, our explainer on how RAG retrieves fresh knowledge covers it in more detail.

The full RAG architecture

The six steps above assume the documents are already indexed. A complete system has two pipelines: one that prepares documents ahead of time, and one that answers queries on demand.

Preparing documents for search

Before anyone can ask a question, each document must be turned into something searchable:

• Upload. The PDF lands in storage, for example an AWS S3 bucket.
• Load. A document loader reads the file and extracts its text into the pipeline.
• Split. A text splitter breaks the text into smaller chunks or pages. This matters because embedding a whole book as one vector would blur all its topics together, and because models have limited context windows.
• Embed. Each chunk is passed through an embedding model and becomes a vector.
• Store. The vectors, along with the text they represent, are saved in a vector database.

At the end of this stage the document is ready to be queried. It typically runs once per upload, not on every question.

Answering a query

When a question arrives, the second pipeline runs:

• Embed the query. The question is converted into a vector using the same embedding model, so it lives in the same space as the stored chunks.
• Search. A similarity search finds the chunks closest to the question.
• Retrieve context. Those chunks are pulled from the vector database.
• Build the prompt. The retrieved text and the user's question are combined into a system prompt.
• Call the model. The finished prompt is sent to the LLM API.
• Respond. The model returns an answer grounded in the retrieved material.

One detail that trips up beginners: the query and the documents must be embedded with the same model. Vectors from two different embedding models are not comparable, and mixing them silently ruins retrieval quality.

What you would otherwise write yourself

Without a framework, a team building this would hand-roll prompt management, retrieval logic, context injection, multi-step chains, memory, tool integrations and the orchestration that ties them together. None of it is conceptually hard, but it adds up, and it tends to couple your code tightly to one model and one database. LangChain's reusable abstractions for each of these let you build faster and change components later with less rewriting.

What the framework brings

Four benefits come up again and again.

Chains as a composition model

A chain links steps, such as a prompt template, a model call and an output parser, into a single workflow you can run, test and reuse. In current versions this is expressed through Runnables and LCEL, which let you pipe components together.

Model-agnostic code

Because LangChain supports the major LLM providers behind a common interface, your application is not welded to one vendor. Switching models becomes a configuration change rather than a rewrite, which is useful both for cost control and for trying newer models.

A broad ecosystem

The framework ships with, or connects to, a large set of components and integrations: loaders for many file types, many vector stores, embedding providers and tools. Much of what you need likely already exists as an integration.

Memory and state

Conversational applications need to remember what was said earlier. LangChain offers ways to manage conversational context, memory and state across interactions. The recommended approach here has shifted across releases, with newer guidance leaning on LangGraph for stateful workflows, so check the current documentation for your version.

What you can build with it

Common application types include:

• Conversational chatbots, where users talk to an AI in natural language.
• Knowledge assistants, which help people find and understand information in their own documents or knowledge bases.
• AI agents, which work through multi-step tasks and decide which tools to use along the way.
• Workflow automation, where an LLM is one step in a larger automated process.
• Summarisation and research helpers, which condense material and support research.

When to consider an alternative

LangChain is one option among several, and each alternative has its own emphasis:

• LlamaIndex focuses heavily on connecting LLMs to external data and building data and RAG applications.
• Haystack targets search, question answering, RAG and agentic applications.
• Semantic Kernel is an open-source SDK from Microsoft that embeds AI models in existing software and coordinates multi-step AI workflows.
• DSPy treats LLM systems as programs to be optimised, rather than asking you to hand-craft every prompt.
• AutoGen is built around applications where several agents collaborate and talk to each other.
• CrewAI is aimed at orchestrating teams of agents working on tasks together.
• PydanticAI is a Python framework for production applications and agents with structured, type-safe outputs.

A rough rule of thumb: if your application is mostly retrieval over your own data, LlamaIndex or Haystack are worth comparing. If you care most about typed outputs, look at PydanticAI. If you want multi-agent collaboration as the central idea, AutoGen or CrewAI fit that framing. LangChain's strength is breadth, which makes it a reasonable default when you are not sure yet which shape your application will take. For a head-to-head look at the two most common choices, see our comparison of LangChain and LlamaIndex.

It is also worth knowing when not to reach for any framework. A single-prompt feature, or a small script with one model call and a hand-written prompt, is often clearer without an abstraction layer. Frameworks pay off as the number of steps and swappable components grows.

The building blocks to learn next

With the big picture in place, the natural next step is the individual components that every LangChain application is assembled from:

• Models, the interfaces for talking to different AI models.
• Prompts, which shape how a model responds to its input.
• Chains, which connect components into workflows.
• Indexes, which connect applications to external knowledge. Newer documentation tends to describe this area in terms of loaders, vector stores and retrievers.
• Memory, which keeps context across interactions.
• Agents, which combine reasoning with tools to carry out tasks.

Understanding what each one is responsible for makes it much easier to read LangChain code and to decide which pieces your own application actually needs.

Key takeaways

• An LLM is one component of an application; loading data, retrieval, prompt assembly, memory and the interface are everything else, and that everything else is what LangChain helps you build.
• Semantic search ranks text by meaning using embeddings, which is why it finds the Paris paragraph for a question about France's capital.
• A RAG system is two pipelines: an offline one that loads, splits, embeds and stores documents, and an online one that embeds the query, retrieves context, builds the prompt and calls the model.
• Always embed queries and documents with the same model, or retrieval quality collapses.
• LangChain's main benefits are chain-based composition, provider independence, a wide integration ecosystem and tools for state and memory.
• Alternatives such as LlamaIndex, Haystack, Semantic Kernel, DSPy, AutoGen, CrewAI and PydanticAI each emphasise something different, and a very simple feature may need no framework at all.