What Broke in a Corrective RAG Pipeline for YouTube Q&A With LangGraph

A corrective RAG system is a retrieval-augmented generation setup with three extra parts compared with the plain version:

1. A judge model that scores whether each retrieved chunk is relevant to the user's question.
2. A query rewriter that reshapes the question to retrieve better.
3. A web search fallback for when nothing retrieved is relevant.

The project behind these notes was a question-answering bot for YouTube videos. You give it a video URL and a question, and it answers. It also copes with videos that have no transcript, which matters because LangChain's YouTube loader only works when a transcript already exists. The flow is: download the video, transcribe it with a speech-to-text model (Whisper), embed the text, store the vectors in ChromaDB, then answer questions through a LangGraph graph.

Because the task involved state tracking, retries and fallbacks, LangGraph suited it better than a LangChain chain or a pile of if-else branches. Working with nodes, edges and a shared state object also exposes how a RAG application really functions. Several mistakes along the way taught more than the successes did, and they're collected below.

Chunking

In any RAG system, chunking is the stage that fails silently when it's done wrong. The common approaches are:

1. Recursive character splitting walks through a list of separators and cuts the text until pieces fit a size limit.
2. Semantic chunking groups sentences by meaning, for instance by scoring sentence similarity with a BERT-style model and merging the closest ones.
3. Document-specific splitting follows the format of the source, and is typical for code.
4. Hierarchical parent-child chunking keeps a wide parent segment and smaller child segments cut from it.

Semantic chunking looked like a natural choice first. Videos cover every topic imaginable, so grouping by meaning seemed sensible. Then something odd showed up: for transcripts under about ten minutes, the splitter often returned a single chunk. Digging in explained why. Short videos usually stay on one subject in depth, and semantic chunking has nothing to split when the whole text is about one thing.

Recursive character splitting came next, along with a second mistake: leaving chunk_overlap at zero. The chunks themselves were decent, but answers got worse. Without overlap, a retrieved chunk carried no hint of what came before it, so the generating model lacked context about what the chunk referred to and often blended details together.

The settled approach was recursive splitting with a real overlap, and no semantic chunking for this use case. It handled both short and long videos and produced better transcript chunks. On a small golden set of ten queries with k=5, the pipeline reached a mean reciprocal rank of 0.81. That sample is tiny, so read it as a rough signal and not a benchmark.

Choosing the embedding model

The bot has to handle any video, so the topics span every domain. A domain-specific embedding model can backfire quickly when a video falls outside its training area. For general use, and given modest hardware, the pick was all-MiniLM-L6-v2, a fast, light model that works acceptably almost everywhere.

The catch is an input limit. This model accepts only about 256 tokens, and anything longer gets truncated, which means a long chunk is embedded using only its first 256 tokens. The consequence is silent data loss and uneven embeddings, and it hurt the application's quality here.

The cure is to keep chunk_size below the model's limit. When splitting by characters, stay under the token limit using the rule of thumb that one token is roughly four characters. A splitter set to 800 characters is about 200 tokens, which fits:

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,      # Notice: 800 characters limit here (~200 tokens)
    chunk_overlap=200,
    length_function=len,
    is_separator_regex=False,
)
docs = text_splitter.create_documents([text])
sentence_transformer = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2"
)
vector_store = Chroma(
    collection_name=get_config("vectorDB_collection_name"),
    embedding_function=sentence_transformer,
)

With the two most common retrieval problems out of the way, the next ones concern LangGraph itself.

LangGraph state and ChromaDB

Every node in a LangGraph app shares data through a state object. Each node receives the state and updates it. There are two common ways to define it:

1. TypedDict, a type-hint schema for dictionaries. It gives static checking but no validation at runtime.
2. Pydantic, which validates data at runtime and raises errors on type mismatches.

When the state is a Pydantic BaseModel, every attribute gets validated. Put a complex object such as a ChromaDB client into it and an error appears immediately. Setting arbitrary_types_allowed=True in the state class lets custom types through without complaint.

A deeper problem remains. LangGraph features such as checkpointers, persistent memory and LangSmith observability sometimes serialise and copy the state. Simple types serialise fine, but a ChromaDB instance can trigger a serialisation error. Serialisation just means converting data into a form that can be stored or sent.

The safer design keeps ChromaDB outside the state entirely. It's also cleaner architecture, since a database resource shouldn't be tangled up with ordinary data. Nodes that need the database can still reach it through the config argument passed to aupdate_state():

app.aupdate_state(
    config={
        "configurable": {
            "thread_id": thread_id,
            "recursion_limit": 30,
            "vector_store": vector_store,      # Pass the Vector DB here
        }
    }
)

Async execution and the input() trap

One big advantage of LangGraph is asynchronous execution. It offers async counterparts of the sync calls: ainvoke() instead of invoke() and aupdate_state() instead of update_state(). With them you can write async nodes and a more responsive app.

The easiest way to wreck that is calling input() inside a node. It blocks execution while waiting for the user, so even a fully async application loses the benefit of being async.

The fix is collecting user input outside the nodes and sending it into the graph via aupdate_state():

await app.aupdate_state(
    config={
        "configurable": {"thread_id": thread_id, "recursion_limit": 30}
    },  # adjust as needed
    values={
        "user_query": user_query  # Send the user inputs using the 'values' property
    },
)

Local or cloud transcription

Transcription is the entry point of the whole project, and it became a bottleneck. The first attempt ran a local speech-to-text model, whisper-small, with two problems:

1. Accuracy. It's a reasonable model for local setups with light hardware demands, but its accuracy is only moderate compared with other options.
2. Latency. On limited resources, transcript creation took very long. The real time factor on the test machine was about 1.03, meaning transcription took as long as the video lasted. For videos over five minutes, users waited that long before getting a first answer.

Real time factor is transcription time divided by audio duration. Since Q&A is blocked until the transcript exists, any value above 1 hurts the experience.

The data wasn't confidential, so moving to a cloud service through the Groq API was easy to justify. It solved both problems. Accuracy improved because Groq serves whisper-large-v3-turbo, which beats whisper-small in most cases. Latency improved because stronger cloud hardware brought the real time factor down to about 0.15.

That choice depended on the data being non-sensitive. For private or confidential material, the decision would look very different.

A quick checklist

The lessons above condense into a few checks worth running before trusting a pipeline like this:

• Print how many chunks a typical short transcript produces. A count of one is a warning sign, whatever the splitter promises.
• Confirm that the splitter's chunk size, converted to tokens, sits comfortably under the embedding model's input limit. Truncation never raises an error, so it has to be checked deliberately.
• Read a few retrieved chunks on their own. If one makes no sense without its neighbours, the overlap is too small.
• Keep clients and connections such as the vector store out of graph state, and pass them through configuration.
• Search the graph nodes for blocking calls like input() or synchronous network requests before blaming the framework for sluggishness.
• Measure the real time factor of transcription on the machine that will actually run it, and decide whether the data allows a hosted service.

What's left out

These notes cover only some of the problems met in a first LangGraph application. Other topics deserve their own treatment: system prompts, evaluating the system and each component, building REST endpoints, and deploying to the cloud.