Stop Your RAG From Serving Stale Documents: Recency and Staleness Tracking

Suppose a RAG system already parses PDFs, chunks them, embeds them, retrieves context and even keeps conversation memory across turns. A quieter problem can still sit inside it. The vector index treats every chunk the same. A firmware guide from 2022 and a hotfix bulletin from last Tuesday carry equal retrieval weight as long as both match the query semantically. The agent has no notion that a document was superseded or that a newer version should win.

In production this produces real damage: outdated compliance procedures, deprecated API references, old troubleshooting steps that no longer apply. The agent delivers stale information with full confidence, because nothing told it otherwise.

Fixing this takes three pieces, all of which fit into a Databricks pipeline with Vector Search and Delta Sync.

Three problems at three layers

They look like one issue, but they're separate challenges at different points in the pipeline:

1. Ingestion of changes. New and edited documents have to reach the index without forcing a complete rebuild.
2. Staleness. Each chunk needs a recorded state of current, outdated or superseded, so the system can tell them apart.
3. Ranking. When two chunks are about equally relevant, the newer one should come out ahead.

Problem 1: getting new content in

With Databricks Vector Search on Delta Sync, the infrastructure for incremental updates already exists. Delta Sync follows the Change Data Feed of the source table and pushes inserts, updates and deletes into the index.

The main choice is the pipeline type. In triggered mode you decide when to sync:

# Manual control — you decide when to sync
vsc.create_delta_sync_index(
    ...
    pipeline_type="TRIGGERED",
)

# After appending new chunks:
index = vsc.get_index(index_name="<YOUR_INDEX>")
index.sync()  # Only processes CDF deltas, not full re-index

In continuous mode the index updates itself within minutes of table changes:

# Automatic — syncs within minutes of table changes
vsc.create_delta_sync_index(
    ...
    pipeline_type="CONTINUOUS",
)
# No manual sync() calls needed

For most production setups, triggered mode gives more control. You can ingest a batch of documents and sync once, instead of provoking an index update for every row.

The ingestion flow for new or changed documents looks like this:

from pyspark.sql import functions as F
from pyspark.sql.functions import expr

def ingest_new_documents(volume_path, chunked_table, parsed_table, doc_version="2.0"):
    """Parse new PDFs, chunk them, append to the chunks table.
    Existing chunks from updated docs are soft-deleted first."""

    # Read new files - binaryFile gives us modificationTime
    new_docs_df = spark.read.format("binaryFile").load(volume_path)

    # Check which files already exist in the table
    existing_paths = (
        spark.read.table(chunked_table)
        .filter("is_active = true")
        .select("path").distinct().collect()
    )
    existing_set = {row["path"] for row in existing_paths}

    all_paths = [r["path"] for r in new_docs_df.select("path").distinct().collect()]
    updated_paths = [p for p in all_paths if p in existing_set]

    # Soft-delete chunks from updated documents
    if updated_paths:
        path_list = ", ".join([f"'{p}'" for p in updated_paths])
        spark.sql(f"""
            UPDATE {chunked_table}
            SET is_active = false, superseded_by = '{doc_version}'
            WHERE path IN ({path_list}) AND is_active = true
        """)

    # Parse with ai_parse_document v2.0
    parsed_df = new_docs_df.withColumn(
        "parsed_content",
        expr(f"""ai_parse_document(content, map(
            "version", "2.0",
            "imageOutputPath", "{volume_path}/parsed_images/"
        ))""")
    ).drop("content")

    # ... your chunking logic here (RecursiveCharacterTextSplitter) ...
    # produces df_chunks with [path, chunk]

    # Enrich with ACTUAL file modification time (not current_timestamp!)
    mod_times_df = parsed_df.select("path", "modificationTime").distinct()

    df_chunks_enriched = (
        df_chunks
        .join(mod_times_df, on="path", how="left")
        .withColumn("id", F.monotonically_increasing_id()
                    + spark.read.table(chunked_table).count())
        .withColumn("ingested_at", F.current_timestamp())
        .withColumn("source_modified_at", F.col("modificationTime"))
        .withColumn("doc_version", F.lit(doc_version))
        .withColumn("is_active", F.lit(True))
        .withColumn("superseded_by", F.lit(None).cast("string"))
        .drop("modificationTime")
    )

    # Append - CDF picks up the changes automatically
    df_chunks_enriched.write.mode("append").saveAsTable(chunked_table)

    return df_chunks_enriched

Look closely at the source_modified_at column. It isn't current_timestamp(). It's the real file modification time that the binaryFile reader exposes. That difference matters a great deal for recency scoring, since a document edited three months ago should score differently from one edited yesterday, whenever the ingestion pipeline happened to run.

Problem 2: tracking staleness

Pure similarity search has no idea a chunk is out of date. The chunks table needs metadata columns that record the life cycle of each piece of content.

The schema

The table grows from three columns to eight:

id | path | chunk | ingested_at | source_modified_at | doc_version | is_active | superseded_by

• ingested_at records when the chunk entered the pipeline, as an audit trail.
• source_modified_at records when the source file was really modified, and feeds recency scoring.
• doc_version is a version label for the document.
• is_active is a soft-delete flag, where false means excluded from retrieval.
• superseded_by names the newer document that replaced this one.

Adding the columns to an existing table

A table that already exists, such as docs_chunked, can take these columns without being rebuilt, and source_modified_at can be backfilled from the real file timestamps in the parsed-documents table:

parsed_table = f"{catalog}.{schema}.docs_parsed"
chunked_table = f"{catalog}.{schema}.docs_chunked"

# Add columns
spark.sql(f"""
    ALTER TABLE {chunked_table}
    ADD COLUMNS (
        ingested_at TIMESTAMP,
        source_modified_at TIMESTAMP,
        doc_version STRING,
        is_active BOOLEAN,
        superseded_by STRING
    )
""")
# Backfill source_modified_at from actual file timestamps
# docs_parsed has modificationTime from the binaryFile reader
spark.sql(f"""
    MERGE INTO {chunked_table} AS chunks
    USING (
        SELECT DISTINCT path, modificationTime
        FROM {parsed_table}
    ) AS parsed
    ON chunks.path = parsed.path
    WHEN MATCHED AND chunks.ingested_at IS NULL THEN
        UPDATE SET
            chunks.ingested_at = current_timestamp(),
            chunks.source_modified_at = parsed.modificationTime,
            chunks.doc_version = '1.0',
            chunks.is_active = true,
            chunks.superseded_by = null
""")

Life-cycle operations

With those columns in place, managing freshness is plain SQL:

# Soft-delete an outdated document
spark.sql(f"""
    UPDATE {chunked_table}
    SET is_active = false
    WHERE path LIKE '%Firmware_Guide_v5%'
""")

# Supersede v6 with v7
spark.sql(f"""
    UPDATE {chunked_table}
    SET is_active = false, superseded_by = 'Firmware_Guide_v7.pdf'
    WHERE path LIKE '%Firmware_Guide_v6%' AND is_active = true
""")
# Staleness report
display(spark.sql(f"""
    SELECT path, doc_version, source_modified_at,
        datediff(current_date(), source_modified_at) as age_days,
        CASE
            WHEN datediff(current_date(), source_modified_at) < 30
                THEN '🟢 Fresh'
            WHEN datediff(current_date(), source_modified_at) < 90
                THEN '🟡 Aging'
            ELSE '🔴 Stale'
        END as freshness
    FROM {chunked_table}
    WHERE is_active = true
    ORDER BY source_modified_at DESC
"""))

Rebuilding the index

After adding columns, rebuild the index so the new metadata is synced:

vsc.create_delta_sync_index_and_wait(
    endpoint_name="<YOUR_VS_ENDPOINT>",
    index_name="<YOUR_INDEX>",
    source_table_name=chunked_table,
    primary_key="id",
    embedding_source_column="chunk",
    embedding_model_endpoint_name="databricks-gte-large-en",
    pipeline_type="TRIGGERED",
    columns_to_sync=[
        "id", "chunk", "path",
        "ingested_at", "source_modified_at",
        "doc_version", "is_active", "superseded_by"
    ],
)

From here on, Vector Search carries the freshness fields and can exclude inactive rows when a query comes in.

Problem 3: favouring newer content

Here things get interesting. Flags alone leave the ranking among active chunks purely semantic. A compliance document from 2023 and an updated one from last month, both marked active, get identical treatment.

The remedy is to over-fetch, re-rank using recency, and return only the best few.

The scoring

Exponential decay with a configurable half-life does the job:

import math
from datetime import datetime

def recency_score(timestamp_str, half_life_days=90):
    """
    Exponential decay scoring based on content age.
    - Age 0 days → score 1.0
    - Age = half_life → score 0.5
    - Age = 2× half_life → score 0.25
    """
    try:
        if isinstance(timestamp_str, str):
            parsed_ts = datetime.fromisoformat(
                timestamp_str.replace("Z", "+00:00")
            )
        else:
            parsed_ts = timestamp_str
        age_days = (datetime.now(parsed_ts.tzinfo) - parsed_ts).days
        return math.exp(-0.693 * age_days / half_life_days)
    except Exception:
        return 0.5

The half_life_days value is the main tuning knob, and it should follow the domain. Use about 30 days for fast-moving content such as release notes and incident reports, 90 days for moderately changing material like technical docs, and 365 days for slow content such as architecture specs and compliance standards.

A custom retrieval tool

Replace the stock VectorSearchRetrieverTool with a custom LangChain tool that wraps this recency logic:

from langchain.tools import tool
from databricks.vector_search.client import VectorSearchClient

@tool
def knowledge_search(query: str) -> str:
    """Search knowledge base with recency-weighted ranking.
    Newer documents are prioritized over older ones."""

    vsc = VectorSearchClient(disable_notice=True)
    index = vsc.get_index(index_name="<YOUR_INDEX>")

    # Over-fetch: get 15 results for re-ranking
    results = index.similarity_search(
        query_text=query,
        columns=["chunk", "path", "source_modified_at", "doc_version"],
        filters={"is_active": True},  # Only active chunks
        num_results=15,
    )

    data = results.get("result", {}).get("data_array", [])
    if not data:
        return "No relevant documents found."

    # Re-rank: combined = 70% similarity + 30% recency
    scored = []
    for row in data:
        chunk, path, modified_at, version, similarity = (
            row[0], row[1], row[2], row[3], row[4]
        )
        recency = recency_score(modified_at, half_life_days=90)
        combined = similarity * 0.7 + recency * 0.3
        scored.append({
            "chunk": chunk, "path": path,
            "doc_version": version, "combined_score": combined,
        })

    scored.sort(key=lambda x: x["combined_score"], reverse=True)

    # Return top 5 with source attribution
    parts = []
    for r in scored[:5]:
        parts.append(
            f"[Source: {r['path']} | Version: {r['doc_version']}]\n{r['chunk']}"
        )
    return "\n\n---\n\n".join(parts)

Then hand it to the agent:

agent = create_agent(
    model=model,
    tools=[knowledge_search],  # Custom tool instead of VectorSearchRetrieverTool
    system_prompt=system_prompt,
    checkpointer=checkpointer,
)

Configuring it through YAML

Rather than hardcoding the weights, keep them in the agent's config file:

llm_endpoint_name: databricks-gpt-oss-120b
vector_search:
  index_name: <catalog>.<schema>.docs_chunked_index
lakebase:
  host: <your-lakebase-host>
  endpoint: projects/<project>/branches/production/endpoints/primary
  user: <your-email>
retrieval:
  overfetch_count: 15
  final_count: 5
  similarity_weight: 0.7
  recency_weight: 0.3
  half_life_days: 90

Some presets that suit different domains:

# Release notes / changelogs (fast-changing)
retrieval:
  similarity_weight: 0.5
  recency_weight: 0.5
  half_life_days: 30
# Architecture / compliance docs (slow-changing)
retrieval:
  similarity_weight: 0.85
  recency_weight: 0.15
  half_life_days: 365
# Mixed knowledge base (balanced)
retrieval:
  similarity_weight: 0.7
  recency_weight: 0.3
  half_life_days: 90

How the layers fit together

When a new PDF lands in a Unity Catalog volume, ingest_new_documents() reads it with binaryFile, which captures the modification time. ai_parse_document() extracts text, tables and figures. A recursive splitter chunks the text. Older chunks from the same path are soft-deleted by setting is_active = false. New chunks are appended with the real source_modified_at, and index.sync() carries the change through the Change Data Feed.

When a user asks something, the custom tool queries Vector Search with a filter on is_active, fetches 15 candidates, scores each as similarity times 0.7 plus recency times 0.3, and passes the top five to the model together with source and version labels. The model then answers, preferring newer content.

Administrators manage the life cycle with SQL: UPDATE ... SET is_active = false to retire outdated documents, superseded_by to record version chains, and a staleness report to see the age distribution of content.

What changes in practice

Before recency weighting, a query such as "What's the firmware update procedure?" might return chunks from three versions of the firmware guide. All are semantically relevant, but only the latest has the current steps, and the agent could blend v5, v6 and v7 into one answer.

After adding the is_active filter and source_modified_at scoring:

• v5 chunks are inactive and filtered out before retrieval.
• v6 chunks are inactive, marked as superseded by v7, and filtered out.
• v7 chunks are active, have the highest recency score and go to the model.

The agent now answers from the current version and shows exactly which document it used.

Lessons worth keeping

• Use the file's real modification time, not the ingestion time. A document modified three months ago should look three months old, not "just ingested today" because the pipeline ran this morning.
• Soft-delete instead of hard-deleting. Setting is_active = false lets you audit what was removed and when, while hard deletes lose history.
• Tune the half-life to the domain. Thirty days suits incident reports, 90 suits technical docs and 365 suits architecture specs. No universal default exists, since it depends on how fast the content changes.
• Picking the weights is simple. What takes work is getting source_modified_at right and establishing a consistent life cycle: who decides when a document is superseded, how versions are tracked, and when the staleness report runs. The code is simple, and the process around it is what makes it work in production.

The Databricks documentation on Vector Search Delta Sync, ai_parse_document, Change Data Feed and retrieval quality covers the underlying features.