Vector Search in Azure AI Search: HNSW, Profiles and a Python RAG Retriever

Retrieval is the part of a RAG system that decides whether the final answer is grounded or vague. This walk-through covers how vector search works inside Azure AI Search and then builds a small Python service that takes a question, finds the closest text by meaning and assembles a prompt for a language model. One naming note: the service used to be called Azure Cognitive Search, and plenty of older posts and samples still use that name. The concepts are identical.

Two halves of RAG

Retrieval-augmented generation splits into retrieval and generation. When a question arrives, the system searches a knowledge base and pulls out the most relevant passages. It then hands those passages to a language model together with the question, so the answer rests on real source text.

Generation can only be as good as what it is given. If retrieval returns the wrong passages, the model will answer wrongly or evasively, however capable it is. That is why the choice of search technique matters so much.

Why keyword matching falls short

Classic search matches literal words. A query for "car" can miss a document that only says "automobile". Vector search fixes this by working on meaning.

Each piece of text is converted into a list of numbers called an embedding. Texts that mean similar things land near each other in vector space. At query time the question is embedded too, and the service looks for the stored vectors closest to it, a task known as nearest-neighbour search. In Azure AI Search you declare a vector field in the index, store embeddings there, and the service builds a structure that can scan huge numbers of vectors quickly.

Algorithms and profiles in Azure AI Search

The service offers two search algorithms for vectors:

• HNSW (Hierarchical Navigable Small World) is an approximate nearest-neighbour method. It is quick and is the usual pick for production.
• Exhaustive KNN compares the query with every stored vector. It is exact but slower, so it suits small data sets or checking how accurate the approximate method is.

Algorithms are not attached to a field directly. An index carries a vector search configuration with a list of algorithms and a list of profiles, where each profile gives a name to one algorithm choice. A vector field then refers to a profile by name, so identical algorithm settings can be shared across several fields.

The field type is Collection(Edm.Single), a collection of single-precision floats. You must give it a dimension count, and that number has to equal the output size of the embedding model you use.

The shape of the system

Two paths exist. Ingestion runs once, or whenever the data changes, and fills the index. The query path runs on every question. One rule ties them together: both paths must use the same embedding model. Embed documents with one model and questions with another, and the vectors cannot be compared, so the results become meaningless.

Documents are usually stored as chunks rather than whole files. Smaller chunks give sharper retrieval and fit more easily into a prompt. Each chunk becomes one search document holding its text, its embedding and metadata that traces it back to the source. The sample here uses short documents, so each one counts as a single chunk. A production system would add a chunking step, but the index layout stays the same.

Hands-on build

You need an Azure subscription, Python 3.9 or newer and some comfort running scripts. Embeddings are produced locally with a small open model, so no Azure OpenAI deployment is required to follow along. Switching to Azure OpenAI is covered near the end.

Step 1: Create the search service

In the Azure portal, create an Azure AI Search resource. The free tier is enough here. Once it exists, copy two values: the service endpoint, shaped like https://<your-service>.search.windows.net, and an admin key from the Keys section. Treat that key as a secret. In a real project it belongs in an environment variable or Azure Key Vault and never in source code.

Step 2: Install the libraries

Two packages are needed: the Azure Search SDK and sentence-transformers.

pip install azure-search-documents sentence-transformers

Step 3: Define the vector index

The next script builds an index with a vector field, then defines the vector search configuration with the algorithm and profile before creating the index.

from azure.core.credentials import AzureKeyCredential
from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents.indexes.models import (
    SearchIndex,
    SimpleField,
    SearchableField,
    SearchField,
    SearchFieldDataType,
    VectorSearch,
    HnswAlgorithmConfiguration,
    HnswParameters,
    VectorSearchProfile,
)

endpoint = "https://<your-service>.search.windows.net"
admin_key = "<your-admin-key>"
index_name = "rag-docs"
index_client = SearchIndexClient(
    endpoint=endpoint,
    credential=AzureKeyCredential(admin_key),
)
# 1. Define the fields. The embedding field is the vector field.
fields = [
    SimpleField(name="doc_id", type=SearchFieldDataType.String, key=True, filterable=True),
    SearchableField(name="text", type=SearchFieldDataType.String),
    SimpleField(name="source", type=SearchFieldDataType.String, filterable=True),
    SearchField(
        name="embedding",
        type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
        searchable=True,
        vector_search_dimensions=384,             # must match the embedding model
        vector_search_profile_name="my-vector-profile",
    ),
]
# 2. Define the vector search configuration: one algorithm and one profile.
vector_search = VectorSearch(
    algorithms=[
        HnswAlgorithmConfiguration(
            name="my-hnsw",
            parameters=HnswParameters(
                m=4,
                ef_construction=400,
                ef_search=500,
                metric="cosine",
            ),
        )
    ],
    profiles=[
        VectorSearchProfile(
            name="my-vector-profile",
            algorithm_configuration_name="my-hnsw",
        )
    ],
)
# 3. Create (or update) the index.
index = SearchIndex(name=index_name, fields=fields, vector_search=vector_search)
index_client.create_or_update_index(index)
print("Index created:", index_name)

A few choices in that script deserve explanation. The dimension count is 384 because that is what the chosen embedding model emits. The vector field points at a profile named my-vector-profile, which in turn points at an HNSW algorithm called my-hnsw. The distance metric is cosine, a common default for text embeddings.

HNSW exposes tuning parameters, and Azure ships sensible defaults. The documented values are m of 4 for the number of bi-directional links per node, efConstruction of 400 for the candidate list while building the graph, and efSearch of 500 for the candidate list while querying. Raising them improves recall at the cost of latency and build time, so start with the defaults and tune against your own targets.

Step 4: Embed and upload

This script loads the model, creates a tiny knowledge base of five short documents, computes their embeddings and uploads them to the index.

from azure.search.documents import SearchClient
from sentence_transformers import SentenceTransformer

search_client = SearchClient(
    endpoint=endpoint,
    index_name=index_name,
    credential=AzureKeyCredential(admin_key),
)
# Load the embedding model (produces 384-dimensional vectors)
model = SentenceTransformer("all-MiniLM-L6-v2")
# Our small knowledge base
documents = [
    {"doc_id": "1", "source": "billing-faq",
     "text": "You can update your payment method from the account settings page under Billing."},
    {"doc_id": "2", "source": "billing-faq",
     "text": "Refunds are processed within five to seven business days to the original payment method."},
    {"doc_id": "3", "source": "shipping-faq",
     "text": "Standard delivery takes three to five working days within the country."},
    {"doc_id": "4", "source": "account-faq",
     "text": "To reset your password, click the forgot password link on the login screen."},
    {"doc_id": "5", "source": "shipping-faq",
     "text": "International orders may take up to fourteen working days depending on customs."},
]
# Create embeddings for the text of each document
texts = [d["text"] for d in documents]
vectors = model.encode(texts)
# Attach the embedding to each document and upload
to_upload = []
for doc, vector in zip(documents, vectors):
    to_upload.append({
        "doc_id": doc["doc_id"],
        "source": doc["source"],
        "text": doc["text"],
        "embedding": vector.tolist(),
    })
result = search_client.upload_documents(documents=to_upload)
print("Uploaded", len(result), "documents.")

A real project would read from files or a database, chunk the text and push thousands or millions of documents through the same upload_documents call, normally in batches.

Step 5: Query by vector

Retrieval embeds the user's question with the same model and then issues a vector query. A VectorizedQuery object tells the service which vector to use, how many neighbours to return and which field to search.

from azure.search.documents.models import VectorizedQuery

def search(question, k=3):
    # Embed the question with the same model
    query_vector = model.encode([question])[0]
    vector_query = VectorizedQuery(
        vector=query_vector.tolist(),
        k_nearest_neighbors=k,
        fields="embedding",
    )
    results = search_client.search(
        search_text=None,                 # pure vector search
        vector_queries=[vector_query],
        select=["doc_id", "source", "text"],
    )
    output = []
    for r in results:
        output.append({
            "score": r["@search.score"],
            "source": r["source"],
            "text": r["text"],
        })
    return output

for item in search("how do I get my money back", k=3):
    print(round(item["score"], 4), "|", item["source"], "|", item["text"])

Try a question phrased as "get my money back". None of the sample documents contain those words, yet the document about refunds should rank first, because the comparison is about meaning and not keywords. That is exactly the behaviour a RAG system needs.

Step 6: Assemble the prompt

Retrieval is done. The last step takes the hits and builds a prompt. No particular model is called here, since you might use Azure OpenAI or any other provider. The script only prepares the input.

def build_prompt(question, k=3):
    hits = search(question, k=k)
    context = "\n\n".join(f"- {hit['text']}" for hit in hits)

prompt = (
        "You are a support assistant. Use only the context below to answer "
        "the question. If the answer is not in the context, say that you do "
        "not have enough information.\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {question}\n"
        "Answer:"
    )
    return prompt

print(build_prompt("how do I get my money back"))

The resulting prompt holds the question plus the most relevant passages. Sending it to a model completes the retrieval-augmented loop, with Azure AI Search as the vector store and retriever.

Swapping in Azure OpenAI embeddings

Many Azure teams use hosted embedding models such as text-embedding-3-small instead of a local one. The change is small: call the Azure OpenAI client to create each embedding and set the index dimension count to match the model, for example 1536 for text-embedding-ada-002. Everything else in the index and query code remains as is.

from openai import AzureOpenAI

client = AzureOpenAI(
    api_key="<your-azure-openai-key>",
    api_version="2024-10-21",
    azure_endpoint="https://<your-resource>.openai.azure.com/",
)
def get_embedding(text):
    response = client.embeddings.create(
        input=text,
        model="text-embedding-3-small",   # your deployment name
    )
    return response.data[0].embedding

Azure AI Search additionally supports integrated vectorization: the service invokes an Azure OpenAI model on its own, converting text into vectors during indexing and again for each query. That saves you from writing embedding code and suits larger pipelines, but the basic flow above is enough to understand how vector search behaves.

What can go wrong

Three mistakes account for most disappointing first results. The first is a dimension mismatch: if the index says 384 but the model emits 1536 numbers, uploads fail, and the fix is to read the model's output size and set the field accordingly. The second is mixing embedding models between ingestion and querying, which produces results that look plausible but are essentially random. The third is changing models later without rebuilding, because every stored vector then belongs to the old model's space and has to be regenerated and re-uploaded.

A habit that saves time is to keep the model name and dimension count in one configuration value that both the ingestion script and the query function read, so they cannot drift apart.

Checking retrieval quality

Before trusting the retriever, write ten or so realistic questions and note which document should come back for each. Run them through the search function and look at the top three hits. If the right passage is missing, the usual causes are chunks that are too large or too small, a weak embedding model for your domain, or HNSW settings that favour speed over recall. Compare against exhaustive KNN on the same data to see how much recall the approximate search is losing, since KNN gives the exact answer to measure against.

Notes for moving beyond the demo

• Embedding model. It fixes your vector dimension and strongly shapes retrieval quality, so choose it deliberately.
• Chunking. Split long documents into focused passages before indexing.
• Hybrid search. To mix keyword and semantic matching, pass a value for search_text alongside the vector query. The service merges BM25 keyword results with vector results.
• Semantic ranking. The optional semantic ranker re-orders the top results with a language model for better ordering.
• Index size. Vector indexes consume memory, so watch size and consider vector compression or binary vectors if storage becomes a concern.

From here you can add your own documents, chunking, hybrid search and a model of your choice to grow the demo into a full RAG application.