Query-Free Chunk Selection: Using K-Means Where RAG Has No Question to Ask

Some LLM problems come without a name, a framework or much literature. This one goes as follows: a very large pile of text, no user question, and the job of picking which pieces to hand to a model so that every theme is covered and each piece can be traced to its origin.

It is not retrieval-augmented generation, since nothing is being searched for. It is not prompt compression, and it is not summarisation. The research community has not quite labelled it, and the usual tools miss it. What follows lays out the problem, shows why existing techniques fall short, and describes a simple fix that happens to line up with academic ideas while covering a gap those papers skip.

When there is nothing to search for

Anyone who has built with LLMs in the last couple of years knows the RAG loop: a user asks something, the query is embedded, similar chunks come back from a vector store, they go into the prompt, and an answer grounded in them is generated. The pattern is elegant, well documented and supported by every major framework. It also quietly assumes that a query exists.

Often there is none. Frameworks like LangChain and LlamaIndex are organised around query and retrieval, and most papers on context compression start from a query too. Search for ways to choose the best chunks for an LLM and every hit presumes a user question to steer the choice.

Consider a learning application. Students upload course material: textbooks, slide decks, video transcripts, handwritten notes, sometimes hundreds of pages. They do not search for anything. They say, in effect, "here are my courses, make me study materials." The system must then produce structured outputs such as quiz sheets, flashcards, summaries and concept maps. Each output should cover a different topic and cite where it came from, while the total source volume dwarfs any context window.

Three requirements at the same time

Existing approaches each break at least one of three constraints that this use case imposes together.

1. No query signal. Nothing from the user guides selection, so the system has to judge what matters from the data alone.
2. Guaranteed thematic coverage. The result is a set of outputs that jointly span the breadth of the material. Skipping an entire topic is a serious failure; a biology quiz that never touches genetics is useless.
3. Source traceability. Every output must point back to a specific chunk of the original, for example "based on Chapter 3, pages 12 to 14 of the biochemistry textbook." In education that is not optional, because students need to know where to read further.

No existing method meets all three at once. That is the gap.

What the literature offers, and where it stops

Why sending everything fails

The naive route is to pour all the material into the context window. With models accepting 128k to 200k tokens, it is tempting to think that works. It does not. The "Lost in the Middle" study showed a U-shaped performance curve: models handle the start and end of a context well and consistently miss what sits in the middle. Accuracy dropped by more than 30% when key content moved from the edges to the centre, even for models built for long contexts.

So even when everything fits, large portions of the material get ignored. The quiz would cover the first and last chapters and skip the rest. More context is not the same as better context.

Prompt compression answers a different question

A growing body of work shortens inputs by deleting redundant tokens before they reach the model.

Selective Context has a small language model compute the self-information of lexical units, whether tokens, noun phrases or sentences, and drops those carrying little. It needs no query, which meets the first constraint. But it filters within existing text, producing a compressed copy of the same text and not a selection of representative content across topics. It cannot promise coverage, and the altered text loses its direct link to the original chunks, so the second and third constraints fail.

LLMLingua and LLMLingua-2 extend the idea with iterative compression and learned token classifiers, reaching ratios up to 20x. They share the same basic limit: the text is compressed for the machine, not organised for a person. The result is a prompt a model can parse, not a structured set of content that can be traced.

RECOMP trains compressors that can summarise or extract from retrieved documents, in both extractive and abstractive flavours. It is the most sophisticated of the group, but it explicitly needs a query to decide what to keep, so it fails the first constraint.

In short, prompt compression reduces token counts. It does not pick representative content, guarantee topic coverage or keep the source structure.

Graph ranking represents but does not organise

TextRank and LexRank build a similarity graph over sentences and apply PageRank to find the most central ones, meaning those that best stand for the whole document. They need no query, and for two decades they have been the standard for unsupervised extractive summarisation. Their known weakness is redundancy. Similar sentences earn similar centrality, so the top picks often repeat each other in different words. A TextRank digest of a biology textbook could return three sentences on DNA replication and none on protein synthesis, simply because replication is mentioned more often.

That undermines coverage, even if the major themes tend to show up, and because the methods work on sentences, traceability is only partly satisfied.

Clustering comes closest

Clustering-based summarisation, with K-means, HDBSCAN or similar run on sentence embeddings, groups alike sentences into thematic clusters and takes one representative from each. This neatly cures redundancy: each cluster contributes a single sentence and different clusters cover different themes.

The strongest recent work in this family, "Graphs in Clusters," combines K-means with TextRank. Clustering comes first for coverage, then ranking inside each cluster for importance. It matches or beats pure graph-based and pure cluster-based methods on long-document benchmarks such as ArXiv and PubMed.

It is the nearest relative, but it differs in several ways that matter. Its goal is one flat text summary, whereas here the goal is N structured outputs. It tunes K with the elbow method, whereas here K comes from the user's request. It works on sentences, whereas here chunk granularity is needed to keep source metadata. Its clusters are an invisible intermediate step, whereas here they are the deliverables. And it has no notion of mandatory source tracing.

Putting it side by side

Against the three constraints a pattern shows up. RAG keeps sources but needs a query and cannot promise coverage. Selective Context and LLMLingua need no query but drop source structure and cannot ensure coverage. RECOMP needs a query. TextRank and LexRank work without one but give redundant output and weak traceability. Clustering summarisation satisfies the first two constraints, needing no query and ensuring coverage, but works at sentence level, which breaks the chain back to the source.

There is no established method for chunk selection that is query-free, coverage-guaranteed and source-traceable.

Turning clustering around

The solution comes from inverting how clustering is normally used in NLP. In the literature, K is a hyperparameter. Researchers sweep through K=5, K=10, K=50, plot silhouette scores or inertia curves and settle on a "best" value. The clusters are mathematical objects serving the algorithm, and no user ever sees them.

In this setting, K is a product specification. A student asking for 10 quiz sheets sets K to 10. Five summary pages mean K=5, and thirty flashcards mean K=30. Each cluster maps to exactly one output. The cluster is not an intermediate abstraction; it is the deliverable. A quiz on mitosis exists because a cluster of mitosis-related chunks exists in the semantic space of the uploaded content.

Existing methods use clustering to compress text for the machine. This one uses clustering to organise content for the human.

Downstream, that changes everything. Traditionally K is tuned mathematically and the clusters stay hidden, and what follows clustering is generation of a flat summary. Here K comes from the request, each cluster becomes something the user receives, and each cluster triggers one LLM call yielding one output that cites its source.

The five-step pipeline

1. Chunking. Documents are split with standard splitters from LlamaIndex or LangChain, or with custom rules. Each chunk keeps its text, source file, page number, position and any other metadata. A typical size is 500 to 1000 tokens, overlapping neighbours by 10 to 20%, a compromise between keeping ideas whole and keeping clusters fine-grained. Chunks can optionally be enriched with summaries, keywords or entity tags, which helps clustering but is not required.

2. Embedding. Every chunk becomes a dense vector. The model depends on the stack: Sentence-Transformers is a fast, free baseline, OpenAI's text-embedding-3-small is strong across languages, and Mistral Embed or Cohere Embed are competitive options. What matters is that semantically close chunks yield close vectors regardless of vocabulary.

3. K-means. This is the core step. K-means divides the embedding space into precisely K groups, with K set to the number of outputs the user wants:

from sklearn.cluster import KMeans
n_outputs = 10  # User wants 10 quiz sheets
kmeans = KMeans(n_clusters=n_outputs, random_state=42, n_init=10)
kmeans.fit(embeddings)

Three properties follow directly and answer the three constraints. No query is needed, since K-means groups by similarity between chunks and not relevance to an outside signal. Thematic coverage is guaranteed, since each chunk lands in exactly one cluster and every topic in the data is represented. Redundancy disappears, since chunks that say the same thing have similar embeddings, end up together, and contribute a single representative.

4. Representative selection. For each cluster, take the chunk nearest the centroid, which best expresses the cluster's theme:

import numpy as np
selected = []
for i in range(n_outputs):
    cluster_indices = np.where(kmeans.labels_ == i)[0]
    centroid = kmeans.cluster_centers_[i]
    distances = np.linalg.norm(
        embeddings[cluster_indices] - centroid, axis=1
    )
    best_local_idx = np.argmin(distances)
    best_global_idx = cluster_indices[best_local_idx]
    selected.append(chunks[best_global_idx])  # Full chunk with metadata

The chosen chunk carries all its original metadata, so nothing is lost and everything can be cited. For richer outputs, the three nearest chunks per centroid can be used in place of one; the principle holds.

5. Generation. The selected chunks of each cluster go to the LLM with a prompt suited to the output type, and source metadata travels alongside so the model can cite. Cluster 1 yields Quiz 1, citing pages 12 to 14 of the first PDF. Cluster 2 yields Quiz 2, citing a video transcript at 14:30. The pattern repeats up to cluster K.

When no count is given

Not every user asks for ten quizzes; some simply want "a quiz" or "a summary." Three fallbacks work. A token budget divides the model's effective context by the average chunk size: for a 16k-token model and 800-token chunks, K lands near 20. The elbow method tries several K values and picks the point where inertia stops dropping meaningfully. A rough rule of one cluster per 5 to 10 pages of source also works. In practice the token budget is the most dependable, because it maps directly onto what the model can handle.

What the method guarantees

These are mathematical properties of K-means, not empirical observations.

• Coverage. Every point is assigned to a cluster, so no chunk is orphaned. If a topic exists, it belongs to a cluster with a representative. Top-K retrieval, by contrast, can return K chunks about one popular topic and miss others entirely.
• No redundancy. Similar chunks group together and only one represents each group. Two chunks saying the same thing in different words share a cluster, and only one survives.
• Adjustable granularity. A low K gives broad clusters over wide topics, a high K gives narrow ones over specific concepts, and the user steers this through the request.
• Source preservation. Working on chunks, not tokens or sentences, keeps the full metadata chain on every selected piece. Token-level compression cannot offer this by design.
• Speed. K-means on 500 embeddings of 384 dimensions takes under a second. No LLM call occurs during selection, only at final generation, so the added latency is negligible.

Where it falls short

Spherical assumptions. K-means expects roughly round clusters, and embedding spaces are not always so tidy. A topic with a complex, non-convex shape may be split across clusters. HDBSCAN or Gaussian Mixture Models could help, though they do not guarantee exactly K clusters.

Typical, not necessarily informative. The chunk nearest the centroid is the most typical one. Running TextRank inside each cluster, as the Graphs in Clusters work does, would give a more nuanced pick, and is a promising improvement.

Small topics get squeezed. If the material is 95% biology and 5% chemistry, chemistry will probably get one cluster while biology gets K-1. The chemistry cluster exists but is proportionally compressed. A weighted or stratified scheme could address it.

Chunk quality decides everything. If chunking splits a concept in half or mixes unrelated topics, clustering inherits the damage. Good chunking, with semantic splitting, overlap and metadata enrichment, is not optional.

Comparison in one pass

RAG is fast and keeps sources, but needs a query and may return K chunks on a single topic. Selective Context and LLMLingua are quick and query-free, yet they discard source structure and cannot ensure coverage. RECOMP is powerful but query-dependent. TextRank and LexRank are fast and query-free, yet coverage is partial because of redundancy and traceability stops at the sentence. Graphs in Clusters is query-free with good coverage, but works on sentences and drops source metadata. MapReduce summarisation needs no query either, though coverage is partial, source links vanish through the summarisation steps, and it is slow because it takes several LLM passes.

The K-means chunk selection described here is the only option that runs without a query, guarantees thematic coverage by construction, keeps full traceability at chunk level, finishes in under a second, and needs neither fine-tuned models nor a vector database.

Naming the problem

What makes this hard is less technical depth than the lack of a name. Search for RAG and thousands of results appear. Search for prompt compression and LLMLingua turns up. Search for extractive summarisation and TextRank, LexRank and BERT-based methods appear. Search for how to choose the best chunks from a large corpus, without a query, to maximise thematic coverage while keeping source traceability, and almost nothing comes back.

A fitting label is query-free semantic compression. It is not retrieval, because there is no query. It is not summarisation, because no new text is generated at this stage. It is not prompt compression, because no tokens are removed from existing text. It is selection with guaranteed coverage: choosing the smallest set of chunks that represents the widest span of the material while holding on to the link back to the documents.

The problem arises wherever a system turns large user-supplied content into structured outputs without explicit search intent. Examples include study materials from course content, reports from big document collections, meeting summaries from hours of transcripts, review digests from large pull requests, and legal analysis, where coverage across clauses matters more than answering one question.

The K-means approach is one solution, arguably the simplest. Hybrids that pair clustering with graph-based ranking, information-density scoring or a light LLM pass for topic detection could do better, and the area is wide open.

The whole thing in code

The complete selection pipeline fits in under 50 lines:

from sentence_transformers import SentenceTransformer
from sklearn.cluster import KMeans
import numpy as np
def select_representative_chunks(chunks, n_outputs, model_name='all-MiniLM-L6-v2'):
    """
    Select the most representative chunks from a large corpus
    using K-means clustering. One cluster = one output.

    Args:
        chunks: list of dicts with 'text', 'source', 'page', etc.
        n_outputs: number of outputs the user wants (= K)
        model_name: embedding model to use

    Returns:
        list of selected chunks with cluster metadata
    """
    # Embed
    model = SentenceTransformer(model_name)
    embeddings = model.encode([c['text'] for c in chunks], show_progress_bar=True)

    # Cluster
    kmeans = KMeans(n_clusters=n_outputs, random_state=42, n_init=10)
    kmeans.fit(embeddings)

    # Select representatives
    selected = []
    for i in range(n_outputs):
        cluster_indices = np.where(kmeans.labels_ == i)[0]
        centroid = kmeans.cluster_centers_[i]
        distances = np.linalg.norm(embeddings[cluster_indices] - centroid, axis=1)
        best_local_idx = np.argmin(distances)
        best_global_idx = cluster_indices[best_local_idx]

        selected.append({
            'cluster': i,
            'chunk': chunks[best_global_idx],
            'distance_to_centroid': float(distances[best_local_idx]),
            'cluster_size': len(cluster_indices),
        })

    return selected

There is no vector database, no retrieval pipeline, no fine-tuned model and no LLM call in the selection step. Embeddings, clustering and centroid selection are all it takes.

Where it ran in practice

The pipeline was built and exercised inside a learning application that generates interactive study materials, such as quizzes, flashcards, summaries and concept maps, from whatever a student uploads: lecture notes, textbooks, video transcripts and slides. It also includes tools adapted for learners with dyslexia, ADHD or other cognitive difficulties, along with gamification. The project is still in active R&D, currently available in French, with an English version planned.

In that setting the K-means layer handles content selection. Clustering 500 chunks takes under a second, it scales well, and thematic coverage holds across very different sources, from dense biology textbooks to scattered lecture recordings.

Why the idea is simple on purpose

Choosing representative chunks from a large corpus, with no query, with thematic coverage and with source traceability, is a more common need than the literature suggests. Anyone building LLM tools for education, report generation, meeting analysis or legal review has probably met a version of it.

The approach is intentionally plain. K-means is decades old and sentence embeddings are standard tooling. The contribution lies in the framing: take the user's intent as K, treat each cluster as a deliverable, and keep the chunk-to-source chain intact across the pipeline.

Room for improvement remains. Clustering could be combined with graph-based ranking, information-density scoring or lightweight LLM passes for topic detection. The centroid pick could give way to something subtler, such as ranking inside clusters, diversity-weighted selection or a small LLM call to choose the most informative chunk. The idea is there to be taken, adapted and improved.