One Retriever, Three Backends: Porting NumPy Search to Chroma, FAISS, Pinecone

The simplest vector store is a NumPy matrix of embeddings plus one line of arithmetic: chunk_vectors @ q_vec scores every chunk against the query, and a sort picks the top results. That is a genuine vector database, just the most minimal one possible. This walkthrough takes that exact retriever and ports it to three real backends, Chroma, FAISS and Pinecone, so you can see precisely what each one adds, which of the many named products (Chroma, FAISS, Qdrant, Weaviate, Milvus, pgvector, Pinecone) fits which situation, and which mistakes quietly ruin relevance.

The local examples need no API keys and cost nothing. Pinecone offers a free serverless starter tier that requires a signup and a key but, at the time of writing, no payment card, so the managed path can be tried for free as well. The code assumes a single script, vdb.py, that you run block by block.

Where a NumPy array stops being enough

A hand-made retriever computes cosine similarity against every stored chunk on every query. With a few hundred chunks that is instantaneous. With a few million, it becomes slow, the whole matrix has to sit in RAM, and every embedding disappears when the process exits. A vector database addresses three gaps:

1. Speed at scale. Rather than an exact O(n) comparison against every vector, it builds an approximate nearest neighbor (ANN) index such as HNSW, which finds almost-nearest vectors in roughly O(log n). You give up a small amount of recall in exchange for a very large speedup.
2. Persistence. Vectors are stored on disk or on a managed server and survive restarts.
3. Metadata and filtering. Each vector can carry attributes like {"source": "doc12", "date": "2026-01"}, and a query can be restricted to them, for example nearest vectors where the source is doc12.

The contrast between the two query paths fits in two lines:

your numpy array:   query ──► compare to ALL n vectors ──► sort ──► top-k     (exact, slow at scale)
a vector database:  query ──► walk HNSW graph ──► ~top-k     (approximate, fast, on disk, filterable)

That is the whole value proposition. Everything that follows is mechanics.

Vocabulary you will keep running into

• Embedding or vector: the list of numbers that encodes meaning; the examples here use all-MiniLM-L6-v2.
• Distance metric: how closeness is measured. Cosine (the angle between vectors) is the usual choice for text; dot product and euclidean (L2) are the alternatives. Use whichever your embedding model was trained for.
• Index: the structure that makes search fast. HNSW, a navigable graph, is the default in most databases; IVF and PQ trade memory against speed in different ways.
• Upsert: insert a vector by ID, or update it if the ID already exists.
• Metadata filtering: narrowing a search by stored attributes.
• Namespace or collection: a partition inside one database so unrelated data never mixes.

Chroma: the natural first upgrade

Chroma runs inside your Python process after a single install, with no server and no Docker. That makes it the most direct step up from a NumPy array. Install it together with the embedding library:

pip install chromadb sentence-transformers

The script below uses a small corpus describing a fictional note-taking product called Nimbus, with a topic tag per document. It creates an in-memory client, opens a collection configured for cosine distance through hnsw:space, embeds the documents explicitly and adds them with IDs and metadata. Then it embeds a question and asks for the two nearest documents, printing each with its distance. Note the comment on chromadb.Client(): swapping in PersistentClient(path=...) is all it takes to keep data on disk.

# vdb.py
import chromadb
from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("all-MiniLM-L6-v2")

# Same Nimbus corpus from the RAG tutorial
DOCS = [
    "Nimbus Pro costs 8 dollars per month billed annually, with 50 GB storage.",
    "Nimbus offers a 30-day refund policy on all paid plans.",
    "Nimbus Lite (free) includes 1 GB of storage and up to 50 notes.",
    "Nimbus stores notes encrypted at rest with AES-256; E2E encryption is Pro-only.",
    "Live chat support is Pro-only, Mon-Fri 9-6 UTC; free users get 48h email support.",
]
metadatas = [{"topic": t} for t in ["pricing","refund","pricing","security","support"]]

client = chromadb.Client()                       # in-memory; use PersistentClient(path=...) to save to disk
collection = client.create_collection(
    "nimbus", metadata={"hnsw:space": "cosine"}   # choose the distance metric
)

# Embed and add. (↔ your chunk_vectors = embedder.encode(...))
collection.add(
    ids=[f"doc{i}" for i in range(len(DOCS))],
    embeddings=embedder.encode(DOCS).tolist(),
    documents=DOCS,
    metadatas=metadatas,
)

# Query. (↔ your retrieve(): embed query, find nearest)
q = "How much does the paid plan cost?"
res = collection.query(query_embeddings=embedder.encode([q]).tolist(), n_results=2)
for doc, dist in zip(res["documents"][0], res["distances"][0]):
    print(f"[{dist:.3f}] {doc}")

Map this back to the hand-made version and nothing is conceptually new. collection.add replaces encoding the documents and holding the matrix. collection.query replaces the matrix product, the argsort and the top-k slice. What Chroma contributes is the HNSW index, persistence and metadata support around that same idea.

Chroma can also embed text for you with a built-in model if you pass documents= without embeddings=. Embedding explicitly, as above, keeps the mechanics visible and leaves the model choice in your hands, which matters because the same model must be used for both adding and querying.

Filtering on metadata

Filtering is the feature a plain array never had. The next query searches for "data security" but restricts candidates to vectors whose topic is security via the where argument:

res = collection.query(
    query_embeddings=embedder.encode(["data security"]).tolist(),
    n_results=3,
    where={"topic": "security"},   # only search vectors tagged security
)
print(res["documents"][0])

In production this is what powers requests like "similar tickets from this customer" or "documentation updated this year". The filter is applied together with the vector search rather than as a post-processing step, so you still get up to n_results matches that satisfy it. As an exercise, change the filter to where={"topic": "pricing"} and confirm that the refund and security documents never come back.

FAISS: a fast index library, not a database

FAISS, from Meta, is not a database at all. It is a highly optimised nearest-neighbor library with no metadata and no storage layer of its own, meant to be embedded in your application. Many other systems use it internally. Install the CPU build:

pip install faiss-cpu

The code encodes the documents with normalize_embeddings=True and converts them to float32, which FAISS requires. It builds an IndexFlatIP, an inner-product index; because the vectors are normalised to unit length, inner product equals cosine similarity. After adding the vectors, it encodes the query the same way and calls search for the top two, then maps the returned positions back to the original documents.

import faiss, numpy as np

vectors = embedder.encode(DOCS, normalize_embeddings=True).astype("float32")
index = faiss.IndexFlatIP(vectors.shape[1])   # IP = inner product = cosine on normalized vectors
index.add(vectors)                            # ↔ exactly your numpy array, but optimized in C++

q = embedder.encode(["refund window?"], normalize_embeddings=True).astype("float32")
scores, idx = index.search(q, k=2)            # ↔ your argsort top-k
for s, i in zip(scores[0], idx[0]):
    print(f"[{s:.3f}] {DOCS[i]}")

This is almost literally the NumPy retriever: IndexFlatIP is brute-force cosine search, just implemented in optimised C++. The payoff comes when you replace it with an approximate index such as IndexHNSWFlat or IndexIVFFlat, which keep the same interface but search in sub-linear time. Choose FAISS when you want maximum speed inside one process and are prepared to handle persistence and metadata yourself, for instance by keeping a separate mapping from positions to document records.

Pinecone: fully managed and serverless

Everything so far ran in your own process. Pinecone is a managed vector database reachable over the network: there is no server to run, and it scales automatically. It is the option for teams that do not want to operate infrastructure.

Be aware that the Pinecone SDK changed substantially. The older pinecone.init(api_key=, environment=) style no longer exists; you now create a Pinecone client object, and the package is named pinecone, not pinecone-client. Check the current SDK documentation if anything below fails.

The block opens with the install command and then the Python code (split them if you copy it). It creates a client, creates a serverless index named nimbus only if it does not exist yet, with dimension=384 to match MiniLM and metric="cosine", and connects to it. It upserts (id, vector, metadata) tuples, storing the original text in the metadata because Pinecone returns only what you stored. Finally it queries for the top two matches restricted by a Mongo-style $eq filter on topic and prints each score and text. In real code, read the key from an environment variable or secret store instead of placing it in source.

pip install pinecone

from pinecone import Pinecone, ServerlessSpec

pc = Pinecone(api_key="YOUR_API_KEY")          # get a free key at app.pinecone.io

# Create a serverless index (only once). dimension must match your embedder: MiniLM = 384.
if not pc.has_index("nimbus"):
    pc.create_index(
        name="nimbus",
        dimension=384,
        metric="cosine",
        spec=ServerlessSpec(cloud="aws", region="us-east-1"),
    )
index = pc.Index("nimbus")

# Upsert: (id, vector, metadata) tuples
index.upsert(vectors=[
    (f"doc{i}", embedder.encode(d).tolist(), {"text": d, "topic": m["topic"]})
    for i, (d, m) in enumerate(zip(DOCS, metadatas))
])

# Query with metadata filtering
res = index.query(
    vector=embedder.encode("how much is pro?").tolist(),
    top_k=2,
    include_metadata=True,
    filter={"topic": {"$eq": "pricing"}},      # Mongo-style filter operators
)
for match in res["matches"]:
    print(f"[{match['score']:.3f}] {match['metadata']['text']}")

The shape matches Chroma exactly: create an index, upsert IDs, vectors and metadata, then query top-k with an optional filter. The retrieval step is the same one as before, now running on someone else's autoscaling infrastructure behind an API. Pinecone also supports namespaces, which partition a single index, and integrated inference, where upsert_records and search_records embed text for you so you do not run an embedder yourself.

The broader lesson is that moving from Chroma to Pinecone changed only how you connect and call the store, not the RAG logic around it. In a well-structured application the vector store is a swappable component, which is also why a framework method such as LangChain's .as_retriever() looks identical regardless of backend.

Choosing a backend for your constraints

No vector database is best in general; one is best for a given set of constraints. Roughly:

• NumPy array (hand-built): runs in-process; fine below about 10k vectors, for learning, or when you want total control.
• Chroma: embedded database, in-process or as a local server; prototypes and small-to-medium apps, the easiest start.
• FAISS: index library, in-process; maximum speed when you handle storage and metadata yourself.
• pgvector: a Postgres extension inside your existing database; ideal when you already run Postgres and want vectors next to relational data.
• Qdrant: database, self-hosted with Docker or in its cloud; strong filtering, Rust performance, open source, hybrid search.
• Weaviate: database, self-hosted or cloud; built-in modules, hybrid search, a GraphQL API.
• Milvus: database, self-hosted (heavier to operate) or cloud; built for billions of vectors.
• Pinecone: managed only, in its own cloud; zero operations and serverless autoscaling.

The decision that keeps coming back is managed versus self-hosted. Managed services, Pinecone or the cloud editions of the others, remove operational work but bring ongoing cost and put your data on a vendor's servers. Self-hosting Chroma, Qdrant, Weaviate, Milvus or pgvector means running it yourself while keeping control over cost and data residency. A commonly quoted rule of thumb is that self-hosting starts to pay off once the monthly managed invoice climbs into the low hundreds of dollars; under that threshold, the engineering hours spent on operations usually cost more than they save. Treat that as a heuristic, not a law, since your team's operational capacity matters as much as the invoice.

For learning and most side projects, Chroma or pgvector is enough. Pinecone or Qdrant Cloud make sense once infrastructure is something your team would rather not own, and Milvus makes sense when you genuinely operate at billions of vectors. If you are weighing whether you need a dedicated vector store at all, see why production RAG is moving beyond dedicated vector databases.

Mistakes that bite once a database is running

• Dimension mismatch is the most common error. The index dimension must equal the embedder's output size: 384 for MiniLM, 1536 for many OpenAI embedding models. Mixing models in one index is a guaranteed bug.
• Metric mismatch silently destroys relevance, for example using euclidean distance with a model designed for cosine. Match the metric to the model.
• Approximate is not exact. ANN indexes occasionally miss a true nearest neighbor; that is the price of speed. Most databases expose a knob, such as ef_search for HNSW or nprobe for IVF, that raises recall at the cost of latency. For a deeper look at those parameters, see tuning HNSW indexes for production RAG.
• Re-embedding on every restart happens when nothing is persisted, so you pay the embedding cost on each run. Use Chroma's PersistentClient or a server-backed database.
• Filters have their own cost. Heavy metadata filtering can slow queries; index the fields you filter on if your database supports it.

A good exercise is to take an existing NumPy-based RAG script and replace only its retrieval function with the Chroma collection shown earlier. The generation half does not change at all, which demonstrates that retrieval really is a clean, replaceable layer.

Troubleshooting quick reference

• dimension does not match: your vectors are not the size the index expects. Recreate the index with the correct dimension, or use the embedder it was built for.
• Chroma results look random: the metrics are probably mixed, or different embedders were used for adding and querying. Both must match.
• Pinecone has_index or create_index errors: confirm you are on the current SDK (from pinecone import Pinecone) and not the deprecated pinecone.init style.
• FAISS search returns -1 IDs: the index holds fewer vectors than k. Add data or lower k.
• Everything is slow at scale: you are most likely on a flat, exact index. Switch to HNSW or IVF and tune the recall parameter.

Key takeaways

• A vector database is a hand-built retriever made fast, persistent and filterable. Adding vectors is encoding plus storage; querying is cosine top-k.
• The ANN index is the only truly novel piece: by accepting approximate answers, search stays fast as data grows.
• Metadata filtering and persistence are the production features a plain array lacks.
• When retrieval quality is poor, first decide whether the problem is the embeddings (wrong model or metric) or the index (recall set too low). The returned distances usually tell you which.
• Keep the store behind a narrow interface so switching from Chroma to Pinecone, or anything else, touches only connection and call code.