A Practical RAG Architecture on Firebase and Vertex AI with Firestore Vectors

Retrieval-Augmented Generation has become a go-to pattern for AI apps that deal with private, shifting or specialized information. Rather than expecting an LLM to recall everything from training, RAG lets the model consult an external knowledge source at query time. That makes it possible to build assistants for company documentation, customer records, product knowledge, historical data or internal processes.

RAG is more than a vector database wired to an LLM. A production-grade build combines document processing, embeddings, vector search, metadata filtering, context assembly, prompt design, generation, caching, error handling and monitoring. Below is a practical architecture on Firebase and Vertex AI, with the implementation patterns that matter when a prototype heads toward production.

Three layers

A RAG system has three layers: a knowledge store, a retrieval layer and a generation layer. Source documents or app data are turned into embeddings and saved beside their original text and metadata. When a user asks something, the question gets embedded too, and a semantic similarity search finds the closest pieces of information. Those pieces become the context handed to the language model, which writes the answer from what was retrieved.

The key idea is that the model never has to memorize the whole knowledge base. Retrieval runs fresh for each query, so the underlying data can change with no retraining.

User Question
      |
      v
Generate Query Embedding
      |
      v
Metadata Filtering
      |
      v
Vector Similarity Search
      |
      v
Top-K Relevant Documents
      |
      v
Context Construction
      |
      v
LLM / Gemini
      |
      v
Grounded Response + Sources

The split also helps maintenance. The retrieval strategy can change while the generation model stays, or the LLM can be swapped without rebuilding the knowledge base.

Embeddings

Embeddings underpin semantic retrieval. Rather than treating text as keywords, an embedding model maps it to a numeric vector that captures meaning-level relationships. The reference implementation calls Vertex AI's gemini-embedding-001, with different task types for documents and for queries, since stored documents and incoming questions play different roles.

const { PredictionServiceClient, helpers } =
  require('@google-cloud/aiplatform');
const PROJECT_ID = process.env.PROJECT_ID;
const client = new PredictionServiceClient({
  apiEndpoint: 'aiplatform.googleapis.com'
});
async function generateEmbedding(
  text,
  taskType = 'RETRIEVAL_DOCUMENT'
) {
  const endpoint =
    `projects/${PROJECT_ID}/locations/global/` +
    `publishers/google/models/gemini-embedding-001`;
  const instance = {
    content: text,
    task_type: taskType
  };
  const request = {
    endpoint,
    instances: [helpers.toValue(instance)]
  };
  const [response] = await client.predict(request);
  return response.predictions[0].embeddings.values;
}

Indexing the knowledge base calls for RETRIEVAL_DOCUMENT, and processing a user's question calls for RETRIEVAL_QUERY. Staying consistent here matters, because retrieval quality leans heavily on how both sides are represented.

Batching and concurrency

Early RAG code often embeds each document one at a time. That's fine for a small prototype but wasteful once thousands or millions of chunks need processing. Grouping texts into batches, within request and token limits, works better.

The reference code estimates tokens simply and caps both the number of segments and the estimated tokens per request. It also uses limited concurrency, so several batches run in parallel without swamping the API.

const EMBEDDING_CONFIG = {
  maxSegmentsPerRequest: 100,
  maxTokensPerRequest: 18000,
  concurrency: 3,
  tokenEstimateDivisor: 3
};
function estimateTokens(text) {
  return Math.ceil(
    text.length / EMBEDDING_CONFIG.tokenEstimateDivisor
  );
}
function packIntoBatches(texts) {
  const batches = [];
  let currentBatch = [];
  let currentTokens = 0;
  for (const text of texts) {
    const tokens = estimateTokens(text);
    const exceedsCount =
      currentBatch.length >=
      EMBEDDING_CONFIG.maxSegmentsPerRequest;
    const exceedsTokens =
      currentTokens + tokens >
      EMBEDDING_CONFIG.maxTokensPerRequest;
    if (exceedsCount || exceedsTokens) {
      if (currentBatch.length > 0) {
        batches.push(currentBatch);
      }
      currentBatch = [text];
      currentTokens = tokens;
    } else {
      currentBatch.push(text);
      currentTokens += tokens;
    }
  }
  if (currentBatch.length > 0) {
    batches.push(currentBatch);
  }
  return batches;
}

The limits belong in configuration and shouldn't be baked in as assumptions. In production they should vary by environment, and API latency, rate limits, failure rates and throughput should be watched before concurrency gets tuned.

Keeping vectors beside application data

A handy choice is storing vectors with application data instead of adding another database right away. The reference code keeps document text, embeddings and metadata in Firestore and retrieves similar records through nearest-neighbor vector search.

const { Firestore } = require('@google-cloud/firestore');
const firestore = new Firestore();
async function storeDocumentWithEmbedding(
  collectionPath,
  docId,
  text,
  embedding,
  metadata
) {
  const docRef =
    firestore.doc(`${collectionPath}/${docId}`);
  await docRef.set({
    text,
    embedding,
    ...metadata,
    createdAt: Firestore.FieldValue.serverTimestamp()
  });
}
async function findSimilarDocuments(
  collectionPath,
  queryEmbedding,
  limit = 5
) {
  const collectionRef =
    firestore.collection(collectionPath);
  const vectorQuery = collectionRef.findNearest({
    vectorField: 'embedding',
    queryVector: queryEmbedding,
    limit,
    distanceMeasure: 'DOT_PRODUCT'
  });
  const snapshot = await vectorQuery.get();
  return snapshot.docs.map(doc => ({
    id: doc.id,
    data: doc.data()
  }));
}

Retrieval strategy deserves care. Vector similarity says which documents are semantically near the query, but closeness in meaning doesn't always mean business relevance. As an app grows, metadata therefore becomes very important.

Filtering and tenant isolation

Pure vector search seldom suffices in production. Picture a knowledge base covering several customers, departments, products or regions. A query could resemble documents from multiple tenants, yet the app should return only what the current user may see.

A stronger approach applies structured filters before or alongside semantic retrieval. The reference code filters by topics ahead of the vector query, which narrows the search space and improves relevance.

async function retrieveContext(
  queryText,
  selectedTopics = [],
  limit = 5
) {
  const queryEmbedding =
    await generateEmbedding(
      queryText,
      'RETRIEVAL_QUERY'
    );
let collectionRef =
    firestore.collection('knowledge_base');
  if (selectedTopics.length > 0) {
    collectionRef = collectionRef.where(
      'topics',
      'array-contains-any',
      selectedTopics
    );
  }
  const vectorQuery =
    collectionRef.findNearest({
      vectorField: 'embedding',
      queryVector: queryEmbedding,
      limit,
      distanceMeasure: 'DOT_PRODUCT'
    });
  const snapshot = await vectorQuery.get();
  return snapshot.docs.map(doc => {
    const data = doc.data();
    return {
      text: data.text,
      source: data.source,
      metadata: data.metadata || {}
    };
  });
}

This matters most in multi-tenant apps. Retrieval mustn't turn into an accidental security boundary that exposes another customer's data. Authorization and tenant isolation should be enforced separately from semantic similarity.

Building the prompt

Retrieval solves only half the problem. The retrieved text also has to reach the LLM in a form that distinguishes source material from the user's real question. The reference code builds a dedicated context section, includes source details and tells the model explicitly to answer from the supplied context and to say so when the context lacks what's needed.

async function generateRAGResponse(
  userQuery,
  contextDocuments
) {
  const contextSection =
    contextDocuments
      .map((doc, index) => {
        return `
### Reference ${index + 1}
${doc.text}
Source: ${doc.metadata?.source || 'Unknown'}
`;
      })
      .join('\n');
const prompt = `
You are an AI assistant with access
to a knowledge base.
Use the provided context to answer
the user's question accurately.
CONTEXT:
${contextSection}
USER QUESTION:
${userQuery}
INSTRUCTIONS:
1. Answer using the provided context.
2. If the context is insufficient, say so clearly.
3. Cite the references used.
4. Do not invent information.
ANSWER:
`;
  const result =
    await genAI.models.generateContent({
      model: 'gemini-2.5-flash-lite',
      contents: prompt
    });
  return result.text.trim();
}

Retrieval quality and prompt quality are linked. Strong documents can still produce poor answers when the context is badly organized, duplicated, oversized or presented without clear instructions.

Latency, cost and failures

Latency and cost deserve attention from the start. Embedding generation is an external API call, and the same query may recur often, so caching popular embeddings trims needless requests. Transient errors and rate limits call for retries, not an immediate failure of the whole request. The reference code shows an in-memory embedding cache plus exponential backoff for rate-limit errors.

const embeddingCache = new Map();
async function getCachedEmbedding(text, taskType) {
  const cacheKey = `${taskType}:${text}`;
  if (embeddingCache.has(cacheKey)) {
    return embeddingCache.get(cacheKey);
  }
  const embedding =
    await generateEmbedding(text, taskType);
  embeddingCache.set(cacheKey, embedding);
  return embedding;
}
async function retryWithBackoff(
  fn,
  maxRetries = 3
) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      return await fn();
    } catch (error) {
      if (
        error.code === 429 ||
        error.message.includes('rate limit')
      ) {
        const delay =
          Math.pow(2, attempt) * 1000;
        await new Promise(resolve =>
          setTimeout(resolve, delay)
        );
        continue;
      }
      throw error;
    }
  }
  throw new Error('Max retries exceeded');
}

Once several application instances run, a process-local Map is no longer enough, and a distributed cache gives better consistency across replicas. The principle outweighs the technology: don't recompute costly deterministic work when the result can safely be reused.

Several knowledge sources

Some questions need more than one type of information. An assistant might need a product definition, a practical example and some history to answer well. Pouring all documents into one generic collection makes retrieval less precise.

The reference approach separates definitions, examples and historical data, retrieves each independently and merges the results into a structured context object.

async function retrieveMultiContext(
  queryText,
  options = {}
) {
  const {
    includeDefinitions = true,
    includeExamples = true,
    includeHistorical = false,
    topics = []
  } = options;
const queryEmbedding =
    await generateEmbedding(
      queryText,
      'RETRIEVAL_QUERY'
    );
  const contextPromises = [];
  if (includeDefinitions) {
    contextPromises.push(
      findSimilarDocuments(
        'definitions',
        queryEmbedding,
        3
      ).then(documents => ({
        type: 'definitions',
        documents
      }))
    );
  }
  if (includeExamples) {
    contextPromises.push(
      findSimilarDocuments(
        'examples',
        queryEmbedding,
        5
      ).then(documents => ({
        type: 'examples',
        documents
      }))
    );
  }
  const contexts =
    await Promise.all(contextPromises);
  return contexts;
}

The application gains control this way. Instead of asking one vector search to do everything, it chooses which sources matter for each query and combines them before generation.

Observability

Visibility is a major point where demos and production RAG apps diverge. A reply from the chatbot isn't proof that retrieval is healthy. Teams need figures for retrieval time, documents returned per query, their relevance, cache usage and failure frequency.

The reference code adds a RAGMetrics class that tracks query volume, latency, retrieval accuracy, cache behavior and errors, and computes precision and recall when ground-truth relevant documents exist.

class RAGMetrics {
  constructor() {
    this.metrics = {
      totalQueries: 0,
      averageLatency: 0,
      retrievalAccuracy: [],
      errors: []
    };
  }
logQuery(
    query,
    contextCount,
    latency,
    sources
  ) {
    this.metrics.totalQueries++;
    const previousLatency =
      this.metrics.averageLatency *
      (this.metrics.totalQueries - 1);
    this.metrics.averageLatency =
      (previousLatency + latency) /
      this.metrics.totalQueries;
    console.log({
      query: query.substring(0, 100),
      contextCount,
      latency,
      sourceCount: sources.length,
      timestamp: new Date().toISOString()
    });
  }
  logRetrievalAccuracy(
    retrievedDocs,
    relevantDocs
  ) {
    const retrievedIds =
      new Set(retrievedDocs.map(d => d.id));
    const relevantIds =
      new Set(relevantDocs.map(d => d.id));
    const intersection =
      new Set(
        [...retrievedIds]
          .filter(id => relevantIds.has(id))
      );
    const precision =
      intersection.size / retrievedIds.size;
    const recall =
      intersection.size / relevantIds.size;
    this.metrics.retrievalAccuracy.push({
      precision,
      recall
    });
  }
}

Once such numbers exist, improving RAG turns into an engineering task. Variations like chunk size, top-K, metadata filters, the embedding model, prompt layout and reranking can be tried one at a time, with measurements showing whether retrieval and answers actually got better.

The whole pipeline

A production RAG system is a pipeline and not a single model call. Documents get cleaned and chunked, embeddings are produced efficiently, metadata is stored consistently, retrieval respects authorization and the generation layer receives clear grounding instructions. A practical layout looks like this:

DOCUMENT INGESTION
                    |
                    v
          Clean + Split Documents
                    |
                    v
            Generate Embeddings
                    |
                    v
       Firestore + Metadata Storage
                    |
                    |
USER QUERY --------+
                    |
                    v
           Query Embedding
                    |
                    v
       Authorization + Filters
                    |
                    v
          Vector Similarity Search
                    |
                    v
          Relevant Context
                    |
                    v
          Prompt Construction
                    |
                    v
              Gemini / LLM
                    |
                    v
       Answer + Sources + Metrics

Nothing here locks the design in. Later additions such as hybrid keyword-plus-vector search, rerankers, query rewriting, conversation memory, evaluation sets, streamed output, a shared cache and background ingestion all fit without changing the basic RAG idea.

Lessons

The LLM is only one component. A very capable model can't make up for poor chunking, weak retrieval, wrong metadata, missing authorization checks or irrelevant context. In practice, retrieval quality often decides whether the final system feels smart or flaky.

Building incrementally works best: start with ingestion, embeddings, vector search and a simple grounded prompt, and measure before adding complexity. Then layer on metadata filtering, caching, retries, monitoring, evaluation and advanced retrieval according to real bottlenecks.

RAG changes how AI apps get built because a model can work with knowledge that never had to sit inside its training data. Embeddings, semantic retrieval, structured metadata and context-aware generation together can turn an ordinary application into an assistant that understands its own knowledge base.

The best RAG systems aren't always the most elaborate. They're the ones where retrieval can be measured, context is relevant, access is secure, failures are handled and answers trace back to useful sources. Beginning simple and improving with real metrics beats adding complexity before the actual problem is understood.