Practical notes: RAG Explained: How Applications Give AI Access to Their Own

Use this as an operator-facing rebuild of the ideas in “RAG Explained: How Applications Give AI Access to Their Own Data”: clear stages, ordered code slots, and recovery notes that survive a handoff. The Overview stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.

What Is Retrieval-Augmented Generation (RAG)?

For the What Is Retrieval-Augmented Generation stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Cite the passages that actually grounded the answer. Without citations, operators cannot tell hallucination from an indexing gap.

Without RAG:
[User Question] ──> [LLM] ──> [Answer based on generic training data]

With RAG:
[User Question] ──> [Search Engine / Vector DB]
                          │
                          └──> [Relevant Documents]
                                     │
[User Question] + [Relevant Documents] ──> [LLM] ──> [Factual Answer]

Why RAG Matters: The Limitations It Solves

For the Why RAG Matters The stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Cite the passages that actually grounded the answer. Without citations, operators cannot tell hallucination from an indexing gap.

How Does RAG Work Behind the Scenes?

For the How Does RAG Work stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Cite the passages that actually grounded the answer. Without citations, operators cannot tell hallucination from an indexing gap. For the How Does RAG Work stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.

1. Ingestion Pipeline (Offline / Background):
[Raw Documents] ──> [Chunking] ──> [Embedding Model] ──> [Vector Database]

2. Query Pipeline (Runtime):
[User Question] ──> [Embedding Model] ──> [Vector Search in DB]
                                                  │
                                                  ▼
[User Question] + [Top Matching Chunks] ──> [LLM] ──> [Final Answer]

1. Chunking

When working through the 1 Chunking stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

2. Embeddings

When working through the 2 Embeddings stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

3. Vector Storage

When working through the 3 Vector Storage stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface. When working through the 3 Vector Storage stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.

4. Vector Search (Cosine Similarity)

The 4 Vector Search Cosine stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.

5. Prompt Augmentation

The 5 Prompt Augmentation stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Budget tokens per turn and per session. Agentic tools expand context aggressively; hard caps keep demos from becoming surprise invoices.

Practical Implementation in Node.js

The Practical Implementation in Node stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move. The Practical Implementation in Node stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.

Step 1: Ingesting and Vectorizing Documents

For the Step 1 Ingesting and stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Cite the passages that actually grounded the answer. Without citations, operators cannot tell hallucination from an indexing gap.

// ingest.js
import { OpenAI } from "openai";

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

// In a real application, you would load these from Markdown files or a CMS
const documents = [
  {
    id: "doc_1",
    text: "Enterprise customers can request a full refund within 30 days of contract signing. Contact enterprise-support@example.com."
  },
  {
    id: "doc_2",
    text: "Monthly self-serve subscriptions are non-refundable once the billing cycle begins. Users can cancel anytime to avoid future charges."
  },
  {
    id: "doc_3",
    text: "Custom engineering work and onboarding packages are strictly non-refundable once the kickoff meeting has taken place."
  }
];

async function createEmbedding(text) {
  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: text,
  });
  return response.data[0].embedding;
}

export async function buildKnowledgeBase() {
  const embeddedDocs = [];
  for (const doc of documents) {
    const vector = await createEmbedding(doc.text);
    embeddedDocs.push({ ...doc, vector });
  }
  return embeddedDocs;
}

Step 2: Finding Relevant Context

For the Step 2 Finding Relevant stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Cite the passages that actually grounded the answer. Without citations, operators cannot tell hallucination from an indexing gap.

// search.js
function cosineSimilarity(vecA, vecB) {
  let dotProduct = 0;
  let normA = 0;
  let normB = 0;
  for (let i = 0; i < vecA.length; i++) {
    dotProduct += vecA[i] * vecB[i];
    normA += vecA[i] * vecA[i];
    normB += vecB[i] * vecB[i];
  }
  return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
}

export function findTopMatches(queryVector, knowledgeBase, topK = 2) {
  return knowledgeBase
    .map(doc => ({
      ...doc,
      score: cosineSimilarity(queryVector, doc.vector)
    }))
    .sort((a, b) => b.score - a.score)
    .slice(0, topK);
}

Step 3: Augmenting the Prompt and Generating an Answer

For the Step 3 Augmenting the stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Prefer structured outputs with schema validation over free-form prose when the next step is code or a tool call. For the Step 3 Augmenting the stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.

// answer.js
import { OpenAI } from "openai";
import { createEmbedding, buildKnowledgeBase } from "./ingest.js";
import { findTopMatches } from "./search.js";

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

async function askKnowledgeBase(userQuestion, knowledgeBase) {
  // 1. Convert user question into an embedding
  const questionVector = await createEmbedding(userQuestion);

  // 2. Retrieve top matching chunks
  const matches = findTopMatches(questionVector, knowledgeBase, 1);
  const contextText = matches.map(m => m.text).join("\n---\n");

  // 3. Augment prompt with retrieved context
  const systemPrompt = `
You are a helpful customer service assistant.
Answer the user's question using ONLY the context provided below.
If the context does not contain the answer, say "I do not have enough information to answer that."

Context:
${contextText}
`;

  // 4. Generate answer
  const completion = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      { role: "system", content: systemPrompt },
      { role: "user", content: userQuestion }
    ],
    temperature: 0.2, // Low temperature minimizes creative liberties
  });

  return completion.choices[0].message.content;
}

// Example usage:
const knowledgeBase = await buildKnowledgeBase();
const answer = await askKnowledgeBase("I signed an enterprise deal 2 weeks ago, can I get my money back?", knowledgeBase);
console.log(answer);
// Output: Yes, enterprise customers can request a full refund within 30 days of contract signing. You can email enterprise-support@example.com to initiate the process.

Common Mistakes Developers Make with RAG

When working through the Common Mistakes Developers Make stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

1. Naive Chunking

When working through the 1 Naive Chunking stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

2. Relying Solely on Vector Search

When working through the 2 Relying Solely on stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface. When working through the 2 Relying Solely on stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.

3. Cramming Too Much Context (The Lost-in-the-Middle Problem)

The 3 Cramming Too Much stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.

When Should You Use RAG vs. Fine-Tuning?

The When Should You Use stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.

Conclusion

The Conclusion stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move. The Conclusion stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.

Operational checklist

When working through the Operational checklist stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest.

Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

Pin dependency versions and record the image digest that ran the demo. Reproducibility beats tribal knowledge.

Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.

Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

Before promoting the stack, freeze versions, capture a golden transcript for the critical path, and confirm rollback steps. Shared environments need rate limits, tenancy checks, and a clear owner for secret rotation. Prefer boring reliability over clever one-off demos.

Batch note for 2c8785c21af6: keep provider keys out of the repo, set a per-session token ceiling, and store transcripts next to the eval fixtures so later model swaps stay comparable.