Home / Articles

Vector RAG vs. Keyword RAG: When to Use Each, and Why Hybrid Wins

Vector RAG vs vectorless RAG: embeddings against BM25, SQL and graph search, how each pipeline works, where each fails, and why hybrid with reranking wins.

Publisher: REACTAPP.TOP6 min read1281 words

This article is published in English.

Sources: Vector RAG vs. Vectorless RAG — Which One Actually Knows Where It Left Its Keys?

RAG gives a language model a reference book to consult before it answers. That beats cramming everything into the weights, which is expensive, goes stale and invites hallucination. The interesting question is how the lookup works. Two families dominate: vector retrieval, which ranks by semantic similarity, and what gets called vectorless retrieval, which relies on classical search such as BM25, keyword matching, graphs and SQL. Either can win, depending on your data.

Vector retrieval

This is what most people picture when they hear "modern RAG". Every piece of text becomes a high-dimensional vector (an embedding) and lands in a vector database. At query time the question is embedded too, and the chunks whose vectors sit closest come back. Closeness in vector space stands in for closeness in meaning.

The pipeline has five stages:

  1. Ingest and chunk. Split documents into overlapping chunks, for example 512 tokens with about 10% overlap. Tagging chunks with metadata is optional but worth doing.
  2. Embed. Run each chunk through an embedding model such as OpenAI's text-embedding-3-large, Cohere Embed or BGE to get a dense float vector.
  3. Index. Store the vectors in Pinecone, Weaviate, Qdrant, pgvector or Chroma, indexed with HNSW or IVF so approximate nearest-neighbour search stays fast.
  4. Search. Embed the query, run k-NN or ANN search and take the top-k chunks by cosine similarity.
  5. Generate. Put the retrieved chunks into the prompt and let the LLM produce a grounded answer.

A minimal LangChain version starts with these imports:

# Minimal Vector RAG with LangChain
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.chains import RetrievalQA

and continues with the embeddings, the store and the QA chain:

embeddings  = OpenAIEmbeddings(model="text-embedding-3-large")
vectorstore = Chroma.from_documents(docs, embeddings)
retriever   = vectorstore.as_retriever(search_kwargs={"k": 5})qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    return_source_documents=True
)
result = qa_chain.invoke("What is the refund policy?")

The strength here is semantic understanding. A policy page may promise a "money-back guarantee" while the user types "can I get a refund?" Nothing overlaps word for word, yet the right chunk still comes back. A classic search engine would miss it.

The weakness is that proximity isn't precision. Imagine a chunk that scores 0.94 similar to the query and still misses the exact answer, because the precise term the user wanted never appeared in the text. It feels like a near miss on every query.

Vectorless retrieval

Vectorless is an umbrella term for BM25 (probabilistic keyword ranking, the engine behind Elasticsearch and OpenSearch), SQL lookups over structured data, knowledge graph traversal and hybrid symbolic methods. These have powered enterprise search for decades. They're unglamorous, and they work.

The flow looks like this:

  1. Index. Tokenize the documents and build an inverted index that maps each term to the documents containing it, with TF-IDF-style scoring.
  2. Parse the query. Tokenize it, and optionally expand it with synonyms, stemming or LLM-based rewriting to raise recall.
  3. Rank. BM25 scores documents against the query terms and returns exact or high-overlap matches in order.
  4. Optionally traverse a graph. Link entities, then hop through a structured graph to collect related facts, which suits multi-hop questions.
  5. Generate. Same as before: pass the retrieved passages to the LLM.

A BM25 example with rank_bm25 begins with imports:

# BM25-based Vectorless RAG with rank_bm25
from rank_bm25 import BM25Okapi
import nltk

then tokenizes the corpus, builds the index and queries it:

tokenized_corpus = [nltk.word_tokenize(doc.lower()) for doc in corpus]
bm25 = BM25Okapi(tokenized_corpus)query = "refund policy for subscription plans"
tokenized_q = nltk.word_tokenize(query.lower())
top_docs = bm25.get_top_n(tokenized_q, corpus, n=5)# Inject top_docs into LLM prompt context
context = "\n\n".join(top_docs)

Where vectors capture meaning, this approach captures specificity. Look up a product ID such as SKU-4892-XL and an embedding model will fumble it, while BM25 finds it in milliseconds. Technical documentation, legal clauses, error codes and financial identifiers are its home turf. Picture a detective asking for the exact phrase "crimson ledger": the vector system returns 47 documents on red notebooks, scarlet journals and a strawberry jam recipe, while the BM25 index returns one result, page 14, paragraph 3. For precise lookups of IDs, proper nouns and technical codes, BM25 beats cosine similarity every time.

Side by side

On the core technique, vector search uses dense embeddings with ANN, and the other side uses BM25, keywords, SQL or graphs. Vector search handles paraphrases and synonyms well but is weak on IDs, codes and rare terms. Keyword-style retrieval only matches exact or near-exact terms, yet its scoring is deterministic and very accurate for exact matches.

Operationally, the vector setup is heavier: you need embedding models plus vector database infrastructure, and the whole corpus has to be embedded before you can start. The classical setup is light (Elasticsearch or SQLite) and can index existing documents right away. Typical latency is around 50 to 200 ms for vectors and 5 to 50 ms for classical search. Cost follows the same line, with GPU inference and vector database fees on one side and CPU-friendly search on the other.

For use cases, vectors fit open-domain Q&A, semantic search and chat. Classical retrieval fits legal, medical, financial and code documentation. Hallucination risk is moderate with vectors, since similarity doesn't equal correctness, and lower with exact matching. Multilingual support is strong with multilingual embeddings but needs language-specific tuning on the classical side.

Using both

Most tutorials skip this: production systems usually combine them. The standard recipe is hybrid retrieval with re-ranking. Run BM25 in parallel with vector search, merge the two result lists with Reciprocal Rank Fusion (RRF), pass the top N to a cross-encoder re-ranker, and inject the final ranked chunks into the LLM's context.

That gives you the breadth of semantic search with the precision of keywords. LlamaIndex, LangChain and Haystack all support hybrid retrieval directly, and Cohere's Rerank API and Sentence-Transformers cross-encoders are common choices for the re-ranking stage. Like a heist crew, each method has a specialty: embeddings read the overall shape, BM25 nails the exact codes, and the re-ranker makes sure the model sees only the best five chunks.

Deciding

Three questions settle most cases.

  • Do users phrase things in varied natural language? If yes, lean on vectors. If queries are structured, with exact IDs and codes, lean classical.
  • Is the corpus unstructured (PDFs, articles) or structured (tables, records)? Prose favors vectors. Tables and records favor SQL and BM25.
  • What are your latency and cost limits? A tight budget or strict latency target points to classical retrieval or a light hybrid. A bigger budget allows full hybrid with re-ranking.

Go with vectors for open-ended questions over unstructured prose where semantics matter, such as support bots, research assistants and document Q&A over big corpora. Go with classical retrieval when exact matches are the point (product codes, clause numbers, financial identifiers, error codes, serial numbers) or when cost and latency are hard constraints. In production, use both.

What matters more than picking a side is knowing why each works, so the pipeline matches your data and your users and not whatever the last tutorial used.