Building a Cited Q&A System Over Central Bank PDFs With RAG

Most RAG tutorials run on tidy, manufactured data. Real documents behave differently, and the gap between understanding retrieval-augmented generation and actually shipping it only shows up when someone builds the whole thing against a messy corpus. That was the motivation behind bspQA, a question-answering system over the quarterly Monetary Policy Reports of the Bangko Sentral ng Pilipinas, the Philippine central bank.

The corpus is a good learning target. Each report runs 60 to 70 pages of dense economic analysis, and seventeen of them stretch back several years, all public. The language is domain-specific, there are many files, and the retrieval problem is genuine, since no one can paste seventeen reports into a context window. The goal was modest: ask something like "What was the inflation outlook for 2026?" and receive a grounded, cited answer within seconds, instead of pressing Ctrl+F across a stack of PDFs.

The pattern in one paragraph

RAG ties an LLM's answers to a chosen set of documents instead of whatever it absorbed in training. Documents are turned into vector embeddings ahead of time. When a question arrives, the chunks closest to it are fetched, handed to the model as context, and the model is told to answer from that context alone. The payoff is a system that declines to invent what the documents do not contain.

Two pipelines, kept apart

The design deliberately separates two jobs.

• Ingestion runs offline, once per new document. It downloads PDFs, parses them to plain text, chunks the text, creates embeddings and stores everything in Qdrant.
• Querying runs live on each request. It embeds the question, retrieves relevant chunks and generates the answer. Raw PDFs are never touched at this stage.

Building ingestion

Download step

A manifest.json file is the source of truth. Each report gets an entry holding its URL, a period label and its publication date. The URLs were pasted in by hand, which is as unglamorous as it sounds.

{
  "doc_id": "bsp-mpr-2023-q3",
  "title": "Monetary Policy Report Q3 2023",
  "period_label": "Q3 2023",
  "publication_date": "2023-08-17",
  "url": "https://www.bsp.gov.ph/..."
}

The fetcher skips files already on disk, so rerunning it is harmless, and adding a report means adding one more item to the document list.

Parsing step

Microsoft's MarkItDown converts the PDFs into text. For text-heavy policy reports it does a decent job: headers survive, paragraph breaks come out clean, and tables survive only barely.

MarkItDown is also a heavy dependency that drags in a lot of packages. Because parsing happens only during ingestion and never when answering queries, it lives in an optional dependency group, so the Render-hosted backend does not carry that weight. The general habit is worth adopting early: decide which packages belong to the ingestion environment and which to the serving environment, because the two need not match. A cleanup step also sits between parsing and chunking, and the reason for it comes up below.

Chunking, embedding and upserting

The real decisions start here. Chunks come from a 512-token sliding window with 64 tokens of overlap, counted with tiktoken.

def chunk_text_by_tokens(text, enc, chunk_size=512, overlap=64):
    tokens = enc.encode(text)
    stride = chunk_size - overlap
    chunks = []
    start = 0
    while start < len(tokens):
        window = tokens[start : start + chunk_size]
        chunks.append(enc.decode(window))
        if start + chunk_size >= len(tokens):
            break
        start += stride
    return chunks

Overlap guarantees that a sentence cut at a chunk boundary still appears whole in at least one chunk.

Embeddings come from OpenAI's text-embedding-3-small, which is cheap and solid for dense policy prose. The whole project cost about ten dollars in credits, although the minimum top-up at the time was five, and that goes a long way on a small project.

Every vector gets a UUID5 identifier derived from the source filename and the chunk index. That makes reingestion safe, since a second run overwrites existing records instead of piling up duplicates.

Each stored payload carries the source file, chunk index, the text, the embedding model, the period label, a normalized period key such as q3_2023, and the publication date in ISO 8601. Qdrant holds keyword indexes on the source file and the period key, plus a datetime index on publication date, so metadata filtering stays quick at query time.

Retrieval and the recency trap

A naive setup returns the top K hits by cosine similarity. That works until the corpus spans several years and somebody asks for the current inflation outlook. If the bank used nearly identical wording on inflation in Q1 2022 and Q3 2024, the 2022 chunk might score higher thanks to slightly better phrasing, even though the reader almost certainly wants the latest view.

The remedy is to over-fetch and re-rank. The system pulls three times the requested number of candidates, then re-scores each with a blend of cosine relevance and a normalized recency value.

RECENCY_WEIGHT = 0.85

def combined_score(hit):
    relevance = hit.score / max_score
    recency = (pub_date - min_date).days / date_span_days  # 0.0 to 1.0
    return (1 - RECENCY_WEIGHT) * relevance + RECENCY_WEIGHT * recency

A recency weight of 0.85 means newer documents are strongly favored unless an older one is far more relevant. In a policy corpus where each report effectively supersedes the last, that tradeoff feels right.

Optional filters let a user pin the search to one period or to a date range:

# scope to Q1 2023 only
retrieve_chunks(query, period_label_key="q1_2023")

# scope to a date range
retrieve_chunks(query, publication_date_from="2023-01-01", publication_date_to="2023-12-31")

Without filters the system behaves as a "latest view" assistant. With them it turns into a historical research tool.

Generation

The generator is intentionally plain. Retrieved chunks are formatted as numbered sources and passed to gpt-4o-mini under a system prompt that, paraphrased, says: use only the supplied sources, bring in no outside knowledge, cite every factual claim inline, and say so when the sources do not contain enough.

That last clause deserves serious attention. A tightly constrained prompt is the most practical defense against hallucination. If retrieval returns nothing relevant, the system answers with a fixed fallback message instead of improvising.

API responses carry the answer text plus a structured list of sources, each listing the period it belongs to, when it was published and where the source PDF lives. On the frontend these turn into clickable citations, which keeps every statement traceable.

Problems that showed up in practice

Page furniture polluting chunks

Every page of every report has a header reading Classification: GENERAL and a footer in the form Monetary Policy Report - Q3 2023 | 47. After parsing, these lines appear as ordinary text and get baked into chunks, wasting tokens on noise. The page number in that footer could in fact support more precise citations, an idea that only occurred late.

The fix is a cleaning pass before chunking:

def _clean_parsed_text(text):
    text = text.replace("\x0c", "\n\n")
    text = re.sub(r"Classification:\s*GENERAL\s*\n", "", text)
    text = re.sub(r"Monetary Policy Report\s*[--][^\n]+\|\s*\d+\s*\n", "", text)
    text = re.sub(r"\n{3,}", "\n\n", text)
    return text.strip()

It is obvious in hindsight and easy to miss if the parsed text never gets inspected before embedding. The symptom was odd: during answer testing, the retriever seemed to draw on other chunks instead of the top-ranked one. The footers were splitting paragraphs mid-thought, which quietly altered the context each chunk represented.

An accidental wipe

Qdrant offers a recreate-collection flag that drops and rebuilds the entire collection. Handy in development, catastrophic in production. A guard now ignores the flag whenever the environment looks like production:

def _should_recreate_qdrant_collection():
    if _is_production_environment():
        return False
    explicit = os.environ.get("QDRANT_RECREATE_COLLECTION", "").lower()
    if explicit in ("1", "true", "yes"):
        return True
    app_env = (os.environ.get("ENVIRONMENT") or "").lower()
    return app_env in ("development", "dev", "local")

For a project with a single user, the guard is arguably over-engineered, but it costs almost nothing.

Duplicates after reingestion

After editing a manifest entry and rerunning ingestion, two sets of chunks existed for the same PDF. The solution is to delete a source file's existing chunks before upserting the new ones:

client.delete(
    collection_name=collection,
    points_selector=models.Filter(must=[
        models.FieldCondition(
            key="source_file",
            match=models.MatchValue(value=source_file)
        )
    ]),
)

Together with the stable UUID5 identifiers, reingestion is now idempotent.

Cold starts on the free tier

The backend sits on Render's free tier, which spins down when idle. The first request after a cold start can take 30 to 60 seconds, long enough for a visitor to conclude the app is broken. A light /health endpoint solves most of it: the frontend pings it on page load, waking the server before the first question is submitted. That is one extra HTTP request per session. The UI also carries a short note explaining that the first query may take half a minute to a minute while the backend wakes.

Choosing a chunk size

The value of 512 tokens was found by experiment. Chunks of 128 to 256 tokens were more precise but often lost the context needed to interpret them alone. Chunks of 1024 or more dragged in unrelated surrounding text. The 64-token overlap softens the hard cuts at boundaries and is worth the added storage.

What to change next time

• Build an evaluation set first. An evals directory exists, but no golden question set was written before tuning the recency weight or chunk size, so both choices are educated guesses. A real eval set would ground them.
• Add a reranker. Blending recency into the score is only a heuristic. A cross-encoder such as Cohere Rerank, or a local model, would score query and chunk together and rank relevance more reliably.
• Try hybrid search. Dense vectors can miss exact terms. A phrase like a 7.00% reverse repurchase rate is hard to retrieve if the question words it differently. Pairing BM25 keyword matching with the vector search would close that hole, and Qdrant offers this out of the box.
• Stream responses. gpt-4o-mini takes two to five seconds to answer. Streaming tokens as they arrive makes the interface feel faster even when total time stays the same.

Reusing the design

The architecture is generic. Replace the manifest with another document corpus, update the domain description in the system prompt, and the result is a grounded Q&A system over any set of PDFs. The harder parts, clean chunking, recency-aware retrieval, metadata filtering and production-safe reingestion, are already handled.