How Vector Databases Really Work: From Embeddings to Hybrid Search

There's a phrase that comes up constantly when developers first start working with GenAI systems:

"I get how SQL databases work. But vector databases still feel like a black box to me."

That's a reasonable position to be in.

A conventional database resolves queries through structured relationships:

SELECT * FROM customers WHERE country = 'India';

A vector database is built to answer a fundamentally different kind of question:

"Which stored items carry a meaning closest to this query?"

That shift in what's being asked underlies a surprisingly large portion of today's AI-powered applications.

RAG pipelines, semantic search, recommendation systems, document-based assistants, and autonomous agents all lean heavily on this capability.

What matters most isn't the database technology itself.

It's grasping what a vector actually encodes, how closeness between vectors gets measured, and why picking the right indexing approach matters.

What Exactly Is an Embedding?

Converting meaning into numbers

Consider this sentence:

"Employees can work remotely for up to 30 days."

An embedding model transforms it into a vector:

[0.021, -0.184, 0.731, 0.092, ...]

In practice, embeddings span hundreds or even thousands of dimensions.

Don't fall into thinking of individual numbers as:

"This particular value stands for the word employee."

That's not an accurate mental model.

Instead, the vector is a learned numeric encoding of the semantic characteristics of the text.

Given that, sentences like:

"I love my dog."
"My puppy is my favorite companion."

will typically end up with vectors sitting closer together than a pairing such as:

"I love my dog."
"The database connection timed out."

That's the core mechanism at play.

Meaning gets translated into something you can search mathematically.

Generating an Embedding

Producing an embedding with a model follows a simple pattern:

from openai import OpenAI

client = OpenAI()

response = client.embeddings.create(
    model="text-embedding-3-small",
    input="Employees can work remotely for up to 30 days."
)

vector = response.data[0].embedding
print(len(vector))
print(vector[:5])

The resulting vector isn't meant to be read by a person.

That's fine.

Humans aren't the intended audience for the raw numbers.

The point is to compare this vector against other vectors.

Similarity Is the Real Idea

At its core, a vector database searches through a mathematical space

Say you have three source documents:

A -> Remote work policy
B -> Travel reimbursement policy
C -> Employee leave policy

And your query is:

"Can I work from home while travelling abroad?"

The query gets embedded first.

Then that query vector is compared to each document's vector.

A widely used similarity metric here is cosine similarity:

import numpy as np

def cosine_similarity(a, b):
   return np.dot(a, b) / (
      np.linalg.norm(a) *
      np.linalg.norm(b)
   )

Visually, this looks like:

A larger cosine similarity value tends to indicate that two vectors point in more closely aligned directions.

When vectors are normalized, cosine similarity ends up mathematically close to dot-product similarity.

That overlap explains why both terms show up frequently in discussions of vector search systems.

Why Can't We Just Compare Every Vector?

Scale is what breaks the naive approach

Picture a dataset of:

1,000 documents

At that size, comparing the query against every single vector is trivial.

Now picture:

100 million vectors

Running an exhaustive comparison against every vector at that scale becomes costly.

This is precisely the problem that vector indexing solves.

Rather than checking every vector one by one, approximate nearest-neighbor techniques structure the vector space so lookups for closely matching vectors run far faster.

A well-known family of such techniques is HNSW - Hierarchical Navigable Small World graphs.

You don't need to build HNSW yourself to benefit from vector search.

What you do need to internalize is the underlying trade-off:

A small sacrifice in exact search precision buys you a substantial gain in speed and the ability to scale.

This trade-off sits at the heart of how vector databases operate.

What Does a Vector Index Actually Store?

In a real deployment, each stored record typically holds more than just the embedding:

document = {
    "id": "policy-1042",
    "title": "Remote Work Policy",
    "content": "Employees can work remotely...",
    "department": "HR",
    "country": "India",
    "embedding": vector
}

This detail matters a lot.

The embedding is not a stand-in for the full document.

It functions as an index-friendly representation of the document.

Alongside it, you still need to keep:

• the original text, metadata, unique identifiers, access-control details, and references back to the source

This becomes critical once you're building RAG systems for enterprise use.

Azure AI Search

The point where vector search and enterprise search intersect

Azure AI Search offers vector-based search alongside traditional keyword search and hybrid combinations of the two.

A simplified example of a vector query looks like this:

from azure.search.documents.models import VectorizedQuery

vector_query = VectorizedQuery(
    vector=query_vector,
    k_nearest_neighbors=5,
    fields="content_vector"
)
results = search_client.search(
    search_text=None,
    vector_queries=[vector_query],
    select=["title", "content"]
)

What comes back isn't framed as:

"Here is the mathematically nearest sentence."

Instead, you get a ranked collection of documents, ordered according to whatever vector-search configuration you set up.

That's the point where architectural decisions start to carry real weight.

Why Hybrid Search Often Wins

Meaning-based search and keyword-based search each shine in different situations

Take this question:

"What does policy HR-2026-17 say?"

For this kind of lookup, keyword search performs great.

Now compare it with:

"Can an employee temporarily work from another country?"

Here, semantic search clearly has the edge.

Rather than picking one approach over the other:

Keyword OR Vector

you can combine them:

Keyword + Vector => Hybrid Ranking

Azure AI Search lets you run hybrid queries that blend full-text search with vector search.

This unlocks a useful pattern:

results = search_client.search(
    search_text="remote work from another country",
    vector_queries=[vector_query],
    top=10
)

The specific ranking setup will vary by use case, but the bigger takeaway is this:

Don't push every retrieval problem through embeddings alone.

Metadata Filtering Is Not Optional at Enterprise Scale

Imagine your vector index holds documents like these:

India HR policies
US HR policies
UK HR policies
Finance policies
Engineering documentation

A user then asks:

"What is the India travel reimbursement limit?"

Relying purely on semantic similarity might pull back matching documents from several different regions at once.

Adding metadata filters lets you narrow the scope:

results = search_client.search(
    search_text=query,
    vector_queries=[vector_query],
    filter="country eq 'India'",
    top=5
)

Now the retrieval combines two things:

Semantic relevance + Structured filtering

This is part of why data engineers often pick up enterprise-grade vector search quickly.

It doesn't replace what databases already do well.

Instead, it pairs unstructured semantic retrieval with familiar structured-data thinking.

Pinecone, Weaviate and Databricks Vector Search

Different vendors, same underlying concept

A few platforms you're likely to run into:

Pinecone

A fully managed vector database built primarily for scalable vector search.

Weaviate

An open-source vector database offering vector search, filtering, and a range of extra AI-focused features.

Databricks Vector Search

A vector-search feature built into the Databricks platform, which becomes especially relevant when your enterprise data already sits inside a lakehouse.

The interfaces and operational details vary between these tools.

The underlying concept stays the same:

Don't treat these products as separate technologies to master individually.

Start by understanding the retrieval model itself.

Once you do, each product just becomes a different implementation choice.

Where Vector Databases Actually Make Sense

A vector database isn't the right tool for every AI scenario

Strong candidates include:

Enterprise RAG

Locating relevant policies, documentation and technical knowledge.

Semantic search

Matching on underlying concepts rather than exact keyword matches.

Recommendations

Surfacing products, content or documents that share similar traits.

Support systems

Pulling up past incidents or tickets that resemble the current one.

Code search

Locating functions or snippets that relate semantically to a given problem.

Data engineering assistants

Fetching relevant pipeline docs, schemas, runbooks and incident history.

That said, don't default to a vector database for structured analytical queries.

If the question is:

"What was revenue in Q2?"

and the answer lives inside a governed data warehouse, SQL is typically the more appropriate tool.

Structured question => SQL
Semantic question => Vector Search
Mixed question => SQL + Vector Search

That distinction alone can save a lot of architectural missteps.

The Enterprise Architecture I Prefer

A well-designed retrieval system tends to look more like a pipeline than a single tool.

The vector database is just one piece of that pipeline.

That's probably the single biggest misunderstanding worth correcting.

A vector database on its own doesn't make an AI application smart.

What it provides is an efficient way to retrieve information based on semantic closeness.

The actual intelligence emerges from everything surrounding it:

• the embedding strategy, how content is chunked, the metadata attached, the retrieval logic, ranking decisions, how context gets assembled, evaluation practices, and the model itself

The Mental Model to Remember

If you're a data engineer transitioning into AI engineering, don't start by memorizing product names.

Instead, hold on to this:

Embedding = numerical representation of meaning

Vector Search = find semantically similar representations

Vector Index = make nearest-neighbor search fast

Hybrid Search = semantic + lexical retrieval

Metadata Filter = apply structured constraints

Reranking = improve ordering of retrieved candidates

Once these six concepts feel natural, tools like Azure AI Search, Pinecone, Weaviate and Databricks Vector Search stop seeming like separate mysteries.

They're simply different ways of solving the same underlying question:

Given a query, how do you surface the most useful information from a massive collection of data?

That's ultimately why vector databases matter.

They're not just another entry in the database category.

They're turning into one of the essential retrieval layers powering modern AI systems.

For data engineers, that makes them worth learning properly - not because every single project calls for a vector database, but because more and more, AI applications need a reliable way to find the right information before they can produce the right answer.