Pick RAG Frameworks by Workload, Not by LangChain Momentum

The problem in context

A 2am page is a harsh way to discover that a transitive LangChain dependency shipped a breaking change, pins drifted, and on-call spent an hour bisecting a tree to restore a feature that, at heart, only retrieves chunks and answers over them.

The deeper issue was not one outage. The team could no longer explain its own retrieval path. LangChain had been the day-one default — everyone reaches for it — and abstractions piled up until the system went opaque. Opaque systems fail at night, and they fail slowly, because nobody can point at the broken layer.

This is not a hit piece. LangChain earns its keep when a product truly needs tool calling, memory, prompts, and multi-step orchestration. The failure mode was using a general orchestrator for workloads that were never agentic.

The principle

Choose a RAG framework by workload, not by momentum. Abstractions tax latency, dependency surface, and debuggability. Pay the tax when the problem matches what the framework abstracts; otherwise it is dead weight.

One product can hide two workloads under one brand. Example split: enterprise semantic search over internal docs, and fast document-QA over uploaded PDFs. Neither is an agent. Paying full orchestration cost twice for two dedicated jobs is the smell.

A practical map of tools to jobs looks different once marketing labels are stripped off. Deepset’s Haystack tends to fit explainable production retrieval pipelines. LlamaIndex tends to fit quick private-document QA with a short path from files to answers. Dialog platforms such as Rasa fit intent-heavy support flows. Botpress or Dialogflow fit low-code customer bots. Hugging Face Transformers fit teams that need raw model control or fine-tuning. CrewAI, AutoGen, and DSPy fit multi-agent orchestration experiments.

Those agent-first stacks were reviewed honestly and remain interesting when the product truly needs cooperating agents. Two retrieval workloads, however, meant Haystack and LlamaIndex won on the only axis that mattered for this migration.

Trade-offs

Enterprise semantic search — needs explainable, testable, scalable, self-hostable stacks → Haystack (more moving parts; run a vector DB).

Uploaded-PDF doc-QA — needs fast setup, low latency, low compute → LlamaIndex (narrow on purpose; not an orchestrator).

True multi-tool agent — needs tools, memory, templating → LangChain / CrewAI (latency and dependency tax on the hot path).

Haystack targets modular pipelines with explicit retrieval, routing, and generation, works with real backends (Elasticsearch, OpenSearch, Weaviate), and can be self-hosted for privacy. Pipeline stages stay legible:

from haystack.document_stores import InMemoryDocumentStore
from haystack.nodes import DensePassageRetriever, FARMReader
from haystack.pipelines import ExtractiveQAPipeline

document_store = InMemoryDocumentStore()
retriever = DensePassageRetriever(document_store=document_store)
reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2")pipeline = ExtractiveQAPipeline(reader=reader, retriever=retriever)
response = pipeline.run(query="What is Haystack used for?")

Retriever scores documents; reader answers over top candidates. When something is slow or wrong, inspect a node. Swap InMemoryDocumentStore for OpenSearch without rewriting the rest.

LlamaIndex connects LLMs to local data — ingest, index, query — and stays narrower than LangChain by design:

from llama_index import SimpleDirectoryReader, GPTTreeIndex

documents = SimpleDirectoryReader("<directory_path>").load_data()
index = GPTTreeIndex(documents)
response = index.query("What is the purpose of this document?")

Three lines from a PDF folder to a queryable index. For features that must answer in a couple of seconds, shedding orchestration overhead is a direct latency win.

Measured against a LangChain monolith, a Haystack + LlamaIndex split can look like: lower p95 latency, roughly half the deps on the RAG path, node-level failure clarity, fewer framework-churn incidents, better self-host fit, and hours instead of days to onboard.

How to adopt

Avoid a big-bang rewrite. Stage and measure:

1. Shadow mode — same queries to old and new paths; diff answers and latency with no user impact.
2. Feature-flag cutover — flip traffic in percentages once eval quality matches or beats the incumbent.
3. Carve independent workloads separately — migrate PDF QA alone if it shares no state with search.
4. Delete last — remove the old dependency only after both paths stay green for a release.

An eval harness (fixed questions with known-good answers) turns “is the new path good?” into a number. Shadow mode often reveals mixed results — better passages on some queries, truncated long answers on others — so route long-form to a generative reader and keep extractive for precise lookups. A framework change without measurement is a bet.

Guardrails on the principle: do not cargo-cult this exact split (support bots may want Rasa; raw model work may want Transformers). Do not ban LangChain forever — keep it for the day a genuine multi-tool agent appears.

Where this goes next

Near-term work stays measured: reranking in Haystack, generative readers for long answers, maybe a Rasa intent experiment — tool matched to workload each time. Framework fashion will churn again (CrewAI, AutoGen, DSPy, RAGFlow, Flowise, …). Teams that pick by hype re-litigate the whole stack every season. Teams that name workloads and choose per workload only revisit the piece that changed. If LangChain feels like it is fighting production, the fix is often the right framework for the actual job — not more LangChain.

What “workload-first” changes in org process

Architecture review stops asking “are we standardized on LangChain?” and starts asking “which named workloads exist, and which tool matches each?” That sounds bureaucratic; it is how teams avoid another opaque monolith. Write the workloads down: search over corp wiki, PDF QA for uploads, future multi-tool agent, maybe a support-intent bot. Assign owners and SLOs per workload. Framework choice becomes an implementation detail under each row.

Procurement and security reviews also get easier. Self-hosted Haystack plus OpenSearch is a different data-boundary conversation than a SaaS bot builder. LlamaIndex on ephemeral upload storage is different again. Bundling all three under one “AI platform” ticket obscures those differences.

On-call runbooks should name nodes, not frameworks. “Retriever latency high” is actionable. “LangChain is slow” is not. After the split, pages pointed at OpenSearch query time or reader GPU saturation instead of a dependency mystery.

Training new engineers changes too. Instead of a week of LangChain lore, onboarding can be: here is the Haystack pipeline diagram; here is the LlamaIndex three-step script; here is the eval set; here is how to run shadow mode. Time-to-first-useful-PR drops because the hot path is short.

None of this forbids LangChain. It forbids pretending one orchestrator is the only respectable choice when half the product is not an agent. When a genuine multi-tool assistant finally lands on the roadmap, LangChain or CrewAI can return with a clear job description — and with an eval harness already waiting.

Keep measuring. Keep naming workloads. Keep the hot path short enough that a tired on-call engineer can still explain it at 2am.