RAG Without a Framework: Chunk, Score, Budget, Prompt

RAG is token selection

A model can't know your wiki, your codebase, or yesterday's support tickets. Its weights were frozen at training time, and a gigabyte of text will never squeeze into a 128k-token window. Fine-tuning takes time and money, and it's the wrong tool for facts that shift weekly. That leaves one option, and it's almost embarrassingly direct: search your documents for the few chunks that matter for this question, paste them into the prompt, and tell the model to answer from them.

That's all RAG is: retrieve, then generate. It isn't a new kind of model, it isn't memory, and it isn't magic. It's a way to choose which tokens go into the usual next-token loop. The model still predicts tokens over whatever context you built, and RAG only improves what you built.

Every RAG system, toy or production, boils down to four steps:

• Split documents into chunks.
• Score each chunk against the question.
• Keep the top few that fit the token budget.
• Build a prompt out of those notes plus the question.

Frameworks differ in how they do the scoring step. The shape never changes.

The idea in plain code

The sketch below uses word overlap as a stand-in for real search. It's deliberately primitive, since the point is the structure.

DOCS = [
    "Tokens are subword chunks; a model "
    "sees token IDs, never words.",
    "The context window caps prompt plus "
    "output, measured in tokens.",
    "Temperature rescales the next-token "
    "distribution before sampling.",
    "Ollama serves local models behind an "
    "OpenAI-style API on port 11434.",
]

def score(q: str, doc: str) -> int:
    qw = set(q.lower().split())
    return len(qw & set(doc.lower().split()))

def retrieve(q: str, k: int = 2) -> list:
    ranked = sorted(
        DOCS,
        key=lambda d: score(q, d),
        reverse=True,
    )
    return ranked[:k]

q = "What does the context window cap?"
notes = retrieve(q)
prompt = (
    "Answer using only these notes:\n"
    + "\n".join(f"- {n}" for n in notes)
    + f"\n\nQuestion: {q}"
)
print(prompt)

The string built at the bottom is the part that matters. It holds instructions, then retrieved notes, then the question, and that string is RAG. Pass it to whichever chat client you like and retrieval-augmented generation is done. The "use only these notes" wording matters as well. It's your main protection against a model that would rather answer confidently from its training data than from your documents.

Real systems upgrade exactly one function here, score(). Word overlap misses synonyms, so "laptop" never matches "notebook". Production setups use embeddings, which are vectors that rank by meaning, along with a vector store to search them at scale, and they chunk text more cleverly than one sentence per document. Those are all improvements to the scoring step. The pipeline you just read is the entire architecture.

Checking the budget

Why take the top-k chunks instead of every relevant one? Because the context window is a budget, and retrieval can overdraw it. Text that does fit still isn't free: every token costs money, and a crammed context also makes the model's attention less dependable. Real RAG code therefore counts tokens before it makes the call.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

def fits(prompt: str,         window: int = 8_000,
         reserve: int = 500) -> bool:
    used = len(enc.encode(prompt))
    print(f"prompt uses {used} tokens")
    return used <= window - reserve

if fits(prompt):
    pass  # call the API (Parts 2-5)
else:
    notes = retrieve(q, k=1)  # shrink, retry

That if is what separates a demo from a system. Truncation errors, context that gets quietly dropped, and answers that ignore your documents mostly trace back to skipping this check. Keep a reserve too. The window has to hold the prompt and the output, so leave room for the answer you asked for.

Running the example

pip install -r requirements.txt
python examples/part06_rag_idea.py

The script runs the full toy. It prints retrieval scores for several questions, including one where word overlap fails badly (a good demonstration of why embeddings exist), then shows prompt assembly and the budget gate. It works offline with no key and no API call, because the lesson is the idea. Hooking the assembled prompt up to a live model is a single function call from any basic chat client, and the script prints the spot where that call would go.

What was skipped on purpose

Embeddings, vector stores, chunking strategy and reranking are the machinery that makes retrieve() good, and they belong to a proper RAG implementation, which is where LangChain starts to earn its install. Learning that machinery first has a downside. Engineers who start there tend to see RAG as a product you configure. Those who start with the idea see a dozen lines of control flow wrapped around a search function, and they can inspect every piece when retrieval quality disappoints.

The short version

Strip it down and RAG picks which tokens the model gets to see. Split the documents, rank the pieces against the question, paste the best ones into a prompt that respects the budget, and the usual next-token loop finishes the job. None of that needs a framework, and once you've seen it you'll recognize it inside every framework you meet. The next step is wiring RAG into a real pipeline, using the ReAct and Plan-and-Execute patterns, and seeing where LangChain and LangGraph fit.