Home / Articles

How RAG Gives an LLM Your Private Data: Chunks, Vectors, Prompts

Understand retrieval-augmented generation from ingestion to query, with a small Node.js example, three common failure modes and a RAG versus fine-tuning table.

Publisher: REACTAPP.TOP8 min read1766 words

This article is published in English.

Sources: RAG Explained: How Applications Give AI Access to Their Own Data

Language models write code, explain hard topics and draft essays, yet they stall on anything private. Ask a stock model about an internal product roadmap, a customer database or yesterday's support tickets and it either admits ignorance or, worse, produces a believable fabrication. That false confidence is called hallucination.

A common assumption is that teaching a model private data means fine-tuning, which re-trains its weights on custom datasets. For plain data access in production, fine-tuning is typically the wrong choice. It costs a lot, updates slowly and doesn't reliably stop hallucinations. Production apps use retrieval-augmented generation (RAG) instead, which supplies outside data at request time without retraining. If fine-tuning is weeks of schooling for a student, RAG is an open-book exam with the right page already turned to.

The three words in the name

  • Retrieval means searching your own database or document store for passages related to the question.
  • Augmented means pasting those passages straight into the prompt context.
  • Generation means sending the enlarged prompt to the LLM, which answers from the retrieved facts.

Without RAG the flow is short:

Without RAG:
[User Question] ──> [LLM] ──> [Answer based on generic training data]

With RAG, a search step sits in the middle:

With RAG:
[User Question] ──> [Search Engine / Vector DB]
                          │
                          └──> [Relevant Documents]
                                     │
[User Question] + [Relevant Documents] ──> [LLM] ──> [Factual Answer]

Because the source material travels with the prompt, the model no longer has to recall facts from its frozen parameters.

Why teams choose it

Three obstacles come up when adding AI features to existing systems:

  • Knowledge cutoff. Training data ends on a particular date, so the model can't know what happened this morning or what an API returned two seconds ago.
  • Privacy. Public models can't read an internal wiki, a Notion workspace or a MongoDB database.
  • Auditability. An answer produced from memory can't be traced to a source record. With RAG you know exactly which chunk went into the prompt, so citations are easy.

A frequent misconception is that fine-tuning teaches new facts. It adjusts tone, style or output format. Factual knowledge retrieval is what RAG is for.

Two stages

A production system has an ingestion stage, which prepares files ahead of time, and a query stage, which serves the user in real time. Ingestion runs offline:

1. Ingestion Pipeline (Offline / Background):
[Raw Documents] ──> [Chunking] ──> [Embedding Model] ──> [Vector Database]

The query path runs at request time:

2. Query Pipeline (Runtime):
[User Question] ──> [Embedding Model] ──> [Vector Search in DB]
                                                  │
                                                  ▼
[User Question] + [Top Matching Chunks] ──> [LLM] ──> [Final Answer]

Chunking

A 300-page PDF can't be pushed through one query. Big documents get cut into segments called chunks, commonly 200 to 800 words. A chunk should be small enough to stay on one idea and large enough to keep its context.

Embedding

Computers don't understand words, and keyword matching such as SQL LIKE '%refund%' breaks when people use synonyms. So each chunk goes through an embedding model that returns a list of floating-point numbers, a vector with perhaps 1,536 entries. These vectors place meaning in geometric space. Phrases with similar meaning, such as "How do I cancel?" and "Steps to terminate account", land close together, and unrelated ones like "How to bake sourdough bread" land far away.

The vectors are stored next to the original text in a vector database such as Pinecone, Qdrant, Milvus or PostgreSQL with the pgvector extension.

Retrieval and prompting

When a question comes in, the server embeds it with the same model. The database measures the angle between the question vector and every stored vector (cosine similarity) and returns the three to five closest chunks. The application then inserts them into the system prompt and instructs the LLM to answer using only that material.

A small working example

Imagine a software company that wants an assistant answering questions from its internal refund policy. The ingestion file starts with the OpenAI client import:

// ingest.js
import { OpenAI } from "openai";

and creates the client:

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

A real app would load documents from Markdown or a CMS, and this sample hardcodes a few:

// In a real application, you would load these from Markdown files or a CMS
const documents = [
  {
    id: "doc_1",
    text: "Enterprise customers can request a full refund within 30 days of contract signing. Contact enterprise-support@example.com."
  },
  {
    id: "doc_2",
    text: "Monthly self-serve subscriptions are non-refundable once the billing cycle begins. Users can cancel anytime to avoid future charges."
  },
  {
    id: "doc_3",
    text: "Custom engineering work and onboarding packages are strictly non-refundable once the kickoff meeting has taken place."
  }
];

A helper turns text into an embedding:

async function createEmbedding(text) {
  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: text,
  });
  return response.data[0].embedding;
}

Another builds the knowledge base by embedding every document:

export async function buildKnowledgeBase() {
  const embeddedDocs = [];
  for (const doc of documents) {
    const vector = await createEmbedding(doc.text);
    embeddedDocs.push({ ...doc, vector });
  }
  return embeddedDocs;
}

In production, the vector database does the distance computation over millions of rows in milliseconds. For a demo, cosine similarity can be computed by hand:

// search.js
function cosineSimilarity(vecA, vecB) {
  let dotProduct = 0;
  let normA = 0;
  let normB = 0;
  for (let i = 0; i < vecA.length; i++) {
    dotProduct += vecA[i] * vecB[i];
    normA += vecA[i] * vecA[i];
    normB += vecB[i] * vecB[i];
  }
  return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
}

and the best matches picked out:

export function findTopMatches(queryVector, knowledgeBase, topK = 2) {
  return knowledgeBase
    .map(doc => ({
      ...doc,
      score: cosineSimilarity(queryVector, doc.vector)
    }))
    .sort((a, b) => b.score - a.score)
    .slice(0, topK);
}

Next, one file ties the query, the search results and the LLM together, starting with imports:

// answer.js
import { OpenAI } from "openai";
import { createEmbedding, buildKnowledgeBase } from "./ingest.js";
import { findTopMatches } from "./search.js";

then the client again:

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

The main function first embeds the user's question:

async function askKnowledgeBase(userQuestion, knowledgeBase) {
  // 1. Convert user question into an embedding
  const questionVector = await createEmbedding(userQuestion);

then retrieves the top chunks:

  // 2. Retrieve top matching chunks
  const matches = findTopMatches(questionVector, knowledgeBase, 1);
  const contextText = matches.map(m => m.text).join("\n---\n");

then builds an augmented prompt:

  // 3. Augment prompt with retrieved context
  const systemPrompt = `
You are a helpful customer service assistant.
Answer the user's question using ONLY the context provided below.
If the context does not contain the answer, say "I do not have enough information to answer that."

which ends with the injected context:

Context:
${contextText}
`;

and finally generates the answer:

  // 4. Generate answer
  const completion = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      { role: "system", content: systemPrompt },
      { role: "user", content: userQuestion }
    ],
    temperature: 0.2, // Low temperature minimizes creative liberties
  });

returning the model's text:

  return completion.choices[0].message.content;
}

Usage looks like this, with a question about getting money back after an enterprise deal:

// Example usage:
const knowledgeBase = await buildKnowledgeBase();
const answer = await askKnowledgeBase("I signed an enterprise deal 2 weeks ago, can I get my money back?", knowledgeBase);
console.log(answer);
// Output: Yes, enterprise customers can request a full refund within 30 days of contract signing. You can email enterprise-support@example.com to initiate the process.

The reply didn't come from the model's general training weights. It pulled the precise answer from the small snippet injected at runtime.

Three common mistakes

A short script makes RAG look easy, but real systems fail in three recurring ways.

Chopping text at fixed lengths. Cutting every 500 characters splits words, sentences and tables. If a sentence stating a vital condition is divided across two chunks, both lose their meaning. The fix is to chunk along natural boundaries such as paragraphs, headings or Markdown sections, and to add overlap, for example 500-token chunks sharing 50 tokens.

Relying on vectors alone. Vector search understands concepts but handles exact strings poorly. A search for an invoice number like INV-2026-09 may return invoices that merely look similar. The fix is hybrid search: combine keyword search such as BM25 or full-text with vector search, and merge scores using a method like Reciprocal Rank Fusion (RRF).

Stuffing the prompt. Context windows keep growing, but sending 50 chunks lowers quality. Models recall material at the very start or end of a prompt best, and the middle tends to get lost. Retrieve only the top three to five confident chunks. If more documents must be examined, run a lightweight reranker such as Cohere Rerank before the main LLM.

RAG or fine-tuning

Dimension RAG Fine-tuning
Main goal Supplying factual, changing data Teaching a style, dialect or strict syntax
Updates Instant, by editing database records Slow, needs new data and retraining
Hallucination risk Low, bounded by the supplied context Moderate to high
Citations Built in, since the chunk ID is known Impossible, facts blend into the weights
Upkeep cost Storage and ordinary API calls High GPU training and hosting cost

When data changes daily, weekly or hourly, fine-tuning is impractical, and RAG is the architecture built for dynamic knowledge.

Getting started

The hard part of building with LLMs is factual grounding, not model intelligence. RAG links a static foundation model to private, changing data, and combining a good retrieval layer with an augmented prompt turns an unpredictable text generator into a system that gives verifiable answers. A sensible start: pick a vector store, split documents into sensible chunks, and check that retrieval really finds the right context before blaming the LLM for a poor answer.