Finding Why a Weaviate Query Is Slow With Per-Stage Profiling

Hearing that a query is slow tells an engineer almost nothing. It's a symptom report. Slow relative to what baseline, and in which step? The metadata filter, the vector index, the disk reads that fetch result objects, or the keyword scoring? Until the time is attributed to one of those, every change is a shot in the dark: add memory, adjust ef, lower the limit. Occasionally one of them helps, usually none do, and the real cause stays unknown.

Weaviate has now shipped a feature aimed squarely at that blind spot. Query Profiling reports, stage by stage, where a particular slow query spent its time.

Why the slow query log isn't enough

Slow query logging has existed in Weaviate for a while. Two environment variables get set, the node restarts, and any query crossing a time threshold lands in the logs with timing details. For one purpose that's a good fit: quietly noticing regressions across a whole fleet over weeks and months.

It's a poor tool for the more common moment, when a specific query is slow right now and the cause is needed immediately. Four shortcomings stand out:

• A restart is required, and restarting a node empties the operating system page cache. The first queries afterward read cold data from disk, so the measurement ends up reflecting the restart rather than the query.
• It works retroactively. An entry appears only once a query has already been slow, so the query currently in front of you can't be targeted. The only option is waiting for it to be slow again.
• Logging happens per node. A single query fans out across shards on several machines, and each machine records only its share, which leaves the full picture to be rebuilt by hand.
• It's a temporary switch that has to be remembered and undone. Because it lives outside the deployment configuration, it tends to drift out of sync.

Profiling per query

Query profiling reverses the model. Rather than a global setting that waits for trouble, a request opts in individually, and the timing breakdown comes back inline with the response. No environment variables, no restart, no threshold, and nothing to tidy up afterward.

In Python it takes one import for the metadata class.

from weaviate.classes.query import MetadataQuery

Then a flag on the query itself.

response = collection.query.near_vector(
    near_vector=[0.1, 0.2, 0.3],
    limit=10,
    return_metadata=MetadataQuery(query_profile=True),
)for shard in response.query_profile.shards:
    print(shard.name, shard.node, shard.searches)

The coordinating node gathers profiles from every shard on every node that took part and returns them together, so a single request produces a view of the whole cluster.

Support covers the official Python, JavaScript and TypeScript, Java (v6) and C# clients, as well as raw gRPC. The feature became generally available in Weaviate v1.38. With the flag off, the overhead is one boolean check; with it on, the engine adds timer reads that cost microseconds. It's meant for debugging, so keep it off the hot production path.

Reading the profile

Results are grouped by shard, and stages that never ran are left out. A plain vector search therefore produces a different profile from a BM25 search, and hybrid queries include sections for both. These are the fields worth knowing:

• total_took is the wall-clock time for that shard's search, and every other figure explains part of it.
• objects_took measures reading the final objects from disk, a step known as hydration. A high value signals a disk-bound query, so look at the page cache, the limit and the payload size.
• filters_build_allow_list_took with filters_ids_matched reveal how long the where clause needed to become a list of qualifying IDs, plus the size of that list. A huge list means the filter is broad; a short list that still took long suggests slow inverted-index reads from disk.
• vector_search_took and knn_search_layer_0_took describe HNSW traversal. Since the graph lives in RAM, a large number here means CPU work, not disk. Tune ef, vector dimensionality and the filter strategy before ordering more hardware.
• knn_search_rescore_took shows up only when compression is on. It reads full-precision vectors back to rescore compressed candidates, and that read can reach disk.
• kwd_3_term_time and kwd_4_bmw_time are the two keyword timers to remember. The first covers reading posting lists for each term and responds to term count and disk speed. The second is BlockMax WAND scoring, which grows with long posting lists and weak selectivity.

Three profiles, three different fixes

First case: a shard where the total is a bit under 50 ms.

total_took: 48.2ms
vector_search_took: 8.4ms
filters_build_allow_list_took: 2.1ms
objects_took: 36.8ms

Filtering and searching are cheap here. About three quarters of the time goes to loading objects from disk. The page cache and storage on that node deserve a look, and a smaller limit or lighter payload may help.

Second case: a filter-heavy query.

total_took: 61.5ms
filters_build_allow_list_took: 41.2ms
filters_ids_matched: 12,840,000
objects_took: 3.9ms

Twelve million matched IDs. That's neither a disk problem nor a hardware one. The filter is simply too broad. Narrow it, or apply the wide condition some other way instead of as a pre-filter over everything.

Third case: a compressed vector index.

vector_search_took: 35.1ms
knn_search_layer_0_took: 9.2ms
knn_search_rescore_took: 25.4ms

Walking the graph takes only 9 ms. Nearly all vector time is rescoring, which pulls full-precision vectors back off disk. The remedy sits in storage and compression settings, not in the graph.

The three remedies have nothing in common. Without a breakdown, all three would have arrived as the same complaint: the query is slow.

Closing the visibility gap

Tuning vector database performance has long had a visibility problem. The engine always knew where time went; extracting that knowledge was the difficult part. Query profiling closes the gap with the smallest possible change: one flag, one query, one response. Weaviate's solution engineering team reportedly reaches for it first when a customer reports slowness, since it converts a vague complaint into a named stage with a specific fix.

Teams running Weaviate in production can add query_profile=True to their debugging toolkit. The next time somebody says a query is slow, the answer can be a measured number instead of a guess. The full announcement, titled Query Profiling: See Where a Slow Query Spends Its Time, and the query profiling page in the official docs cover the details.