Evidence-Family Retrieval: Making RAG Context Cover Every Part of a Question

Five relevant chunks can still leave a RAG system without what the whole question needs. The paradox fades once a question turns out to need evidence of several separate kinds. Picture someone asking how app settings, secrets and the permissions granted to workloads should be handled as one setup. A retriever may return five excellent passages about configuration. All are relevant, yet the context is incomplete if it says nothing about credentials or permissions.

This is a built-in weakness of plain top-k retrieval: ranking chunks one at a time says nothing about whether the answer's required concepts are all present.

Relevance isn't completeness

A conventional pipeline ranks chunks and keeps the first k. That's fine when one type of evidence settles the question, and fragile when the question calls for comparison, synthesis or several sources of truth. Suppose the top five chunks all concern Secrets, but the answer also needs ServiceAccount identity and RBAC authorization. The ranking is reasonable locally and incomplete globally.

Raising top_k might help, but not dependably. A bigger context can bring duplicates, noise, latency and cost with no promise that the missing concept shows up, and if one source family dominates the candidates it may fill the first 20 positions on its own. The objective needs a coverage constraint:

Useful context = relevance + required-family coverage + trusted provenance - noise

Evidence-family retrieval adds that constraint while keeping ordinary relevance ranking.

What counts as a family

An evidence family is a set of sources or passages that can meet one obligation in a question. It need not equal one file, topic label, embedding cluster, retrieval strategy or chunk. Picture a question on exposing an application safely: its families might be Ingress routing, Service discovery and readiness behavior. Several documents can qualify for the Service family, and a document may sit in more than one family when its metadata and content support it.

Plan evidence before final selection

For recognized composite question shapes, the project uses a deterministic evidence planner. Conceptually, its output is a set of (source_pattern, rescue_query) pairs, with an entry for every family the question needs:

def evidence_plan(question: str):
    # Returns list of (family_pattern, rescue_query) tuples
    # for recognized composite question shapes
    if asks_about_workload_identity(question):
        return [
            (SECRET_SOURCE_PATTERN,      "Kubernetes Secret credentials Pod"),
            (SERVICE_ACCOUNT_PATTERN,    "Pod ServiceAccount workload identity"),
            (RBAC_SOURCE_PATTERN,        "RBAC least privilege RoleBinding"),
        ]
    return []  # unknown shapes continue through normal retrieval

The scope is deliberately modest, well short of open-ended autonomous planning. Question shapes known to matter get predictable coverage rules, while everything else flows through ordinary retrieval. The property that counts is inspectability: a plan can be examined and tested, so a team that finds a family missing can trace the fault to question detection, source classification, candidate retrieval, reranking, filtering or final truncation.

Recognizing membership

In the case study, family membership comes from matching a regular expression against the chunk's source name:

pattern.search(str(chunk.source))

This is a deterministic source contract, not semantic clustering. What you gain is precision and auditability: a document matches the declared family or it doesn't. What you pay is upkeep, because renamed files and new collections need their mappings updated. A reusable platform should favor explicit metadata over filename patterns:

{
  "source": "security-policy.pdf",
  "document_id": "doc-123",
  "chunk_index": 17,
  "evidence_family": "access-control",
  "authority_tier": "official",
  "version": "2026-07"
}

Membership then reduces to a lookup: chunk.metadata["evidence_family"] == required_family.

Rescuing missing families

After first-pass retrieval, the pipeline counts matching candidates for each planned family. If a family has fewer than the desired number, a targeted sparse search fires with that family's rescue query:

for family_pattern, rescue_query in plan:
    matches = [c for c in candidates if family_pattern.search(c.source)]

    if len(matches) < desired_candidate_count:
        # Targeted rescue — bounded, not an open retry loop
        rescued = sparse_search(rescue_query, top_k=wide_limit)
        candidates.extend(
            c for c in rescued if family_pattern.search(c.source)
        )

Rescue runs only for missing families, using a focused query tied to the family's contract, and it brings back several candidates so the reranker can choose the strongest instead of settling for the first available passage.

Reranking picks passages, not families

Planning decides which families must be represented, and reranking decides which available chunks answer the question best. The two are complementary. The project's reranker blends a weighted mix of signals:

0.4 × normalized cross-encoder score
+ 0.3 × normalized retrieval score
+ 0.3 × lexical overlap
+ conditional exact-token bonus

With candidates ranked, family-aware selection reserves the top-ranked matching chunk for each required family, then fills the remaining slots by global rank:

selected = []

# Step 1: reserve best chunk per required family
for family in required_families:
    best_match = first_ranked_match(family, ranked_chunks)
    if best_match:
        selected.append(best_match)

# Step 2: fill remaining slots by global rank
selected.extend(c for c in ranked_chunks if c not in selected)

final_chunks = selected[:top_k]  # constrained top-k, not an alternative to ranking

Why chunks from one source may need to stay together

An early version of family-aware selection made a mistake by deduplicating on filename. Imagine one NetworkPolicy document with separate passages on default-deny behavior, ingress rules and egress rules, all needed by one composite question. If selection allows only one chunk per filename, two required passages vanish. The right deduplication policy depends on the evidence obligation: it should cut redundancy without destroying coverage, and one source may legitimately supply several complementary passages when each meets a different obligation.

Authority versus relevance

A noisy research paper can resemble a production question semantically, and a generic dashboard guide can share many query terms. Neither should push out an approved operational runbook that directly answers the question. Three signals should stay distinct:

Relevance:  Does this passage discuss the question?
Authority:  Is this an approved source of truth?
Coverage:   Which required evidence obligation does it satisfy?

One score shouldn't quietly stand in for all three. Authority metadata also has to survive ingestion, retrieval, fusion, reranking, caching and API serialization, not just sit in the vector database.

Corrective RAG without losing coverage

Corrective RAG scores retrieved chunks and may retry when evidence looks weak. It sharpens precision but can clash with evidence planning, because a narrow passage that seems only partly relevant on its own may be vital to a multi-document synthesis. If CRAG discards it, a required family disappears from the answer. To prevent that, the implementation regards a validated plan as domain policy: after CRAG filtering it runs a fresh wide search per planned family and picks the strongest authoritative passage. Order matters:

Retrieve broadly
→ shape and rerank candidates
→ reserve family coverage
→ grade evidence (CRAG)
→ restore validated required families
→ build context

A mature design can use several planning levels, chosen by how well the question shape is understood. Deterministic rules suit high-risk or regulated shapes where full auditability matters, and more flexible levels cover broader domains and novel composite questions, always validated against a controlled taxonomy. The pattern also travels. A financial assistant asked whether a transfer is allowed and what approval it needs can't make up for missing account data with five good policy passages, and family planning makes those obligations explicit.

Measuring coverage

Chunk relevance alone can't validate a family-aware retriever. A simple family-recall metric exposes the gap:

Evidence-family recall = covered required families / total required families

# Example: Secret + RBAC covered, ServiceAccount missing
Evidence-family recall = 2/3 = 0.67

# This failure is invisible to standard chunk-relevance metrics

Test suites ought to include rescue of a missing family, survival through reranking and through CRAG, valid alternative sources, complementary chunks from a single document, authority filtering, a top-k lower than the number of required families, and questions that must not trigger any plan.

Failure modes

• A planner that triggers too broadly, because a generic keyword activates an unrelated plan. Use phrase combinations and negative cases.
• An incomplete plan, where declared families are retrieved yet the question still can't be answered. The missing concept belongs in the plan and in regression tests.
• Source renames that make regex membership silently stop matching. Corpus-contract tests should confirm planned families resolve to real sources.
• Rescue queries overfit to evaluation questions. Write them around domain concepts and check with paraphrases and sealed holdouts.
• Reservations that protect weak evidence. Retrieve several candidates per family and rerank before reserving, with authority and minimum-relevance checks still applied.
• Too many families for the context budget, which means the planner must prioritize, merge compatible evidence or ask for clarification.
• A model that ignores part of the evidence. Complete retrieval doesn't guarantee complete synthesis, so prompts and faithfulness checks should reflect the plan.

Next improvements

Several upgrades are worth making next. Put family identity into versioned chunk metadata rather than filenames. Write a domain evidence taxonomy that names owners and authority rules. For unusual composite questions, add a structured planner with tight constraints, and tune the minimum relevance bar separately for each family. Selection could weigh relevance, coverage, redundancy and token budget together. Tracing should expose the plan and family coverage. Beyond recall, measure how precise the planner is, test automatically for corpus drift and renamed sources, and compare rule-based, classifier-based and LLM planners on sealed holdouts. Finally, the system should say so openly when a required family can't be found.

The aim isn't to complicate retrieval for every query. It's to spend extra effort only when a question really needs several kinds of evidence. Top-k asks which chunks look most relevant, and evidence-family retrieval adds a second question: does the final context hold every kind of evidence a complete answer requires? Relevance says whether a chunk is useful, authority says whether to trust it, and coverage says whether the answer can be complete. Production RAG needs all three.