Retrieval by Association: A Memory-Based Mental Model for Vector Search

Years can pass without a single thought about a childhood teacher. Then a whiff of chalk dust, the smell of a school cafeteria or a song that used to play on the bus ride home brings that person back instantly and vividly. Nothing was looked up by name. Something similar arrived, and the memory surfaced on its own.

That is the behaviour semantic search, retrieval-augmented generation (RAG) and AI recommendation systems try to reproduce, imperfectly but seriously, and the component that makes it possible is the vector database. This article uses recall by association as a mental model to explain what vectors are, how similarity is measured, how nearest neighbor search scales, and how the pieces form a retrieval pipeline. It also points out where the analogy stops holding, because those are exactly the places production systems tend to go wrong.

Storing by meaning instead of by name

Human memory has no alphabetical index. There is no mental folder named "Food" containing a subfolder "Italian" containing a file called "Pizza". That strict hierarchy is how a conventional database organises data: every record lives at a known address and is found by an exact key.

Memory is organised by meaning and association instead: by context, by feeling, by how one thing relates to others. The idea of pizza is linked to Friday evenings, a trip to Naples, comfort food, stretching strings of cheese, the scent of oregano, and perhaps a college roommate who ruined every batch of homemade dough. Thinking about pizza does not open a single file. It lights up a neighbourhood of connected memories, the most strongly linked ones first.

A vector database borrows exactly this principle. Items are stored according to what they mean, and they are retrieved by how similar they are to the request, not by whether a key matches exactly.

What a vector encodes

To understand the database you first need to understand what it holds. A vector here is simply an ordered list of numbers that stands for meaning.

A machine has no lived sense of what pizza is. What an embedding model can learn, by processing enormous amounts of text, is the company a word keeps. "Pizza" shows up near "cheese", "Italian", "dough", "oven" and "slice". Those surroundings differ from the ones around "sushi" but overlap heavily with those around "flatbread" or "calzone".

The model condenses those patterns into a fixed-length list of numbers, commonly a few hundred to a couple of thousand values; 384 and 1,536 are typical sizes. No single number carries a readable label, yet together they position the item precisely in a high-dimensional space. The property that matters is this: inputs with similar meanings end up with vectors that sit close together. "Pizza" lands near "flatbread" and very far from "quarterly earnings report".

In other words, meaning is turned into distance. Everything else in this article follows from that single move.

Similarity as a spectrum, not a match

A traditional query is binary: a row either satisfies the condition or it does not. Filter for "dog" and you get rows containing exactly "dog", with nothing for "puppy", "golden retriever" or "loyal four-legged companion".

Semantic similarity replaces that yes-or-no answer with a score that says how close two meanings are. With an illustrative scale, "puppy" might score 0.94 against "dog", "wolf" something like 0.71, and "invoice" around 0.08. The metric is usually cosine similarity or a related distance such as dot product or Euclidean distance, and the right choice depends on how the embedding model was trained.

This mirrors the chalk-dust effect: not an exact hit, but one cue activating nearby memories, which in turn activate their neighbours. In a vector database the mechanics are numerical. The query is converted into a vector, and the database returns stored items whose vectors lie closest to it. Close means similar in meaning.

That is why a search for "how do I fix a slow database query" can surface a document titled "optimizing query performance at scale" even though the two phrases barely share a word. Their vectors are near each other because they express the same intent.

A caveat about the numbers

Similarity scores are relative, not absolute. A 0.8 from one embedding model is not comparable to a 0.8 from another, and even within one model the typical range depends on the domain. Treat scores as a way to rank candidates, and if you apply a cutoff, calibrate it on your own data rather than picking a round number.

A related correction: plain keyword-based SQL cannot express semantic similarity, but that does not mean SQL databases are excluded. Extensions such as pgvector, discussed below, add vector columns and similarity operators to Postgres, so the capability can live inside a relational database.

Nearest neighbor search at scale

How does the database actually locate the closest vectors? The core operation is called nearest neighbor search, and making it fast is the main reason vector databases exist.

Picture every stored item as a star in a galaxy, placed according to its meaning, with related items clustering in the same region. A query drops a new star into that galaxy and asks which five existing stars are nearest.

The naive approach measures the distance from the query to every stored vector, sorts the results and returns the top few. For thousands of vectors that is quick and perfectly adequate. For millions or billions, comparing against everything becomes expensive very quickly.

Production systems therefore rely on approximate nearest neighbor (ANN) algorithms. Instead of visiting every star, they use an index that narrows the search to promising regions, accepting a small loss in accuracy in exchange for a large gain in speed. The most widely used algorithm today is HNSW, short for Hierarchical Navigable Small World. You do not need its internals to use a vector database, but you should know it is the reason a search over a hundred million embeddings can come back in milliseconds.

"Approximate" deserves attention. An ANN index can occasionally miss a true nearest neighbour, and the rate at which it finds the real top results, called recall, depends on index parameters that trade memory and latency for accuracy. If retrieval quality seems inexplicably patchy at scale, index settings are worth checking; our guide to tuning HNSW indexes for production RAG covers those knobs in depth.

Inside the embedding store

At its core, a vector database is storage optimised for one job: holding vectors and running nearest neighbor searches over them very quickly.

Think of a library that ignores the Dewey Decimal System and shelves books by how they feel. Books about grief sit beside books about loss, which sit beside books about loneliness, then isolation, then memoirs of solo expeditions. Nobody assigned those categories by hand; the arrangement emerged because readers drawn to one tend to want the others.

Pinecone, Weaviate, Chroma and Qdrant are examples of that library. You load them with document embeddings, product descriptions, images converted to vectors or patterns of user behaviour, and they maintain the index so that lookups stay fast as the collection grows.

Each stored record generally has three parts:

• An ID that uniquely identifies the item.
• The vector that represents its meaning.
• Optional metadata such as title, date, category or source URL, which can be used to filter results.

Metadata is more valuable than it first appears. A realistic request looks like this: find the five documents most similar to the query, but only those from the last 30 days and only from the engineering team's knowledge base. That is vector search combined with a metadata filter, and most production systems rely on the combination. How the filter is applied matters too: filtering after the similarity search can leave you with fewer results than you asked for, so check whether your database filters during the search itself.

The retrieval pipeline end to end

Whether it sits inside a RAG system, a semantic search feature or any application that makes use of stored knowledge, retrieval follows the same four steps.

1. Embed the content. Every document, article, product description or record you want to be searchable goes through an embedding model, and the resulting vector is stored alongside the original content. Long documents are usually split into chunks first, because one vector for an entire manual blurs too many ideas together.
2. Embed the query with the same model. This is not optional. Different embedding models produce vectors in unrelated spaces, so comparing a query from one model with documents from another yields meaningless distances. Switching models means re-embedding the whole collection.
3. Search. The query vector goes to the database, nearest neighbor search finds the closest stored vectors, and the top results come back, typically with similarity scores.
4. Use the results. In a RAG pipeline the retrieved passages are handed to the language model as context, so the answer is based on real documents rather than guesswork. In a recommender, the results are the recommendations. In a search feature, they are the hits shown to the user.

From the user's point of view, a relevant answer appears within moments. Underneath, meaning was turned into numbers, those numbers were compared across millions of stored items, the closest matches were returned, and a response was built from genuine content.

Why vector search is now everywhere

Not long ago, vector databases were a niche tool reached for only when building semantic search or a specialised recommender. They are now a standard part of many AI applications. RAG needs somewhere to store and query document embeddings. Agents query knowledge bases by meaning. Large-scale recommendation runs on vector similarity. Multimodal search, such as finding images from a text description or products from an uploaded photo, also works through vectors.

If you are building on large language models in production, there is a good chance you already depend on vector search or soon will. The encouraging part is that the core idea is already familiar: memory has been retrieving the nearest associations to whatever just reached your senses for as long as you have been alive. A vector database does the same with numbers, in milliseconds, across millions of items.

Where the memory analogy breaks

The brain comparison is useful for intuition, but a few differences matter in practice:

• Memory adapts continuously; an embedding model is frozen. If your domain vocabulary shifts, the vectors do not update themselves.
• Memory blends context effortlessly; a vector only captures what went into it. Poorly chunked or noisy source text produces poor neighbours.
• Similarity is not relevance. Two passages can be close in meaning while only one actually answers the question, which is why many systems add keyword search or a reranking step on top.

Getting hands-on

You can experiment without building any infrastructure:

• ChromaDB runs locally inside Python with no account, API key or deployment, and takes only a few lines to install and initialise. It suits learning and small projects.
• Qdrant offers a free cloud tier and a tidy Python client, a good fit when you want something close to production without operating servers yourself. Check current tier limits before relying on them.
• pgvector is a Postgres extension. If you already run Postgres, it adds vector search without introducing a separate database, which keeps the stack simple.

Whichever you choose, the workflow is identical: choose an embedding model, embed your content, store the vectors, embed incoming queries and search. The concepts stay the same; only the client library changes.

The same model on both sides of the score

Cosine similarity is the number behind “these two memories feel close”. It only means something when the query and the passage were embedded by the same model. A zero vector, or a mix of two models, quietly ranks noise as a neighbour.

def cosine(a: list[float], b: list[float]) -> float:
    dot = sum(x * y for x, y in zip(a, b))
    na = sum(x * x for x in a) ** 0.5
    nb = sum(y * y for y in b) ** 0.5
    if na == 0 or nb == 0:
        return 0.0
    return dot / (na * nb)

# query and passage must come from the same embedding model
score = cosine(embed(query), embed(passage))

A production query adds a metadata filter before nearest-neighbour search, so one tenant never receives another tenant’s neighbours. The filter is not a reranker. It decides who may enter the candidate set.

hits = collection.query(
    query_embeddings=[embed(query)],
    n_results=8,
    where={"tenant_id": tenant_id},
)

Key takeaways

• An embedding turns meaning into position, so similar items end up close together.
• Similarity scores rank candidates; calibrate any threshold on your own data.
• ANN indexes such as HNSW trade a little recall for large speed gains at scale.
• Metadata filters turn raw similarity into answers that respect time, source and access rules.
• Queries and documents must share one embedding model, and retrieval quality depends as much on chunking and data quality as on the database.