A Web-to-MCP Knowledge Engine: Source Registers, Dual Critics and Privacy Joins

Researchers following the literature, patients matching new studies to their own records, and disability support coordinators or carers tracking fast-moving policy share one daily chore. They need to learn what changed, when it changed and who said it, and they would prefer not to scour the web each morning to find out. The common tools fall short. Keyword alerts treat a vendor blog and an official legal amendment as equals, remember nothing beyond an inbox, and cannot see private records. Search-enabled chatbots summarize whatever ranked highest that morning, lean on the stale hierarchy baked into their training data, forget yesterday's documents, and sometimes miss or fabricate dates and facts.

One response is a system the practitioners call a horizon scan. It works as a knowledge base built from the web, a daily briefing, and a queryable server speaking the Model Context Protocol (MCP). It collects only from trusted, human-reviewed sources, honors subscriptions and licences, and keeps a store that grows with every scheduled run. When evidence for a given day is thin, a critic can decide to publish nothing, leaving only a run record explaining why. The store outlives any single edition and can be queried at any time from a chosen angle against a user's own data, always behind a redacted local privacy boundary. The user's context joins the conversation but is never loaded into the store. Since personal, clinical and case details never enter it, the arrangement fits the Australian Privacy Act 1988 (Cth), while the public store stays shareable with everyone in the field.

The first deployment tracks rules and prices for Australia's National Disability Insurance Scheme (NDIS), a setting in which an incorrect start date or price cap can hurt people. A scheduler fires at 06:00 Sydney time. The editorial desk either publishes or withholds the briefing and records an explicit reason in a run record every time, while the store keeps growing regardless. The brief lands in a Slack channel, and people there can query the knowledge base in place by tagging Claude, which calls the MCP server. One week's brief, for example, reported that participant data may have reached Palantir's analytics platform, next to a Fair Work Commission ruling on a dismissal for breaching the NDIS Code of Conduct.

Dates and links in each edition are pulled from stored facts, and any quotation must match a passage the system has retained. This register is built around the NDIS, but swapping the rows would let the same machinery follow a research field, a health condition or a regulatory body.

Collection driven by a source register

Collection is steered by a source register, a configuration table of monitored endpoints, and not by whatever a search engine ranks highest. Each row describes one source under a fixed schema: publisher name, access method, polling cadence, authority tier, reuse constraint, licensing terms and rules for extracting dates.

Four authority tiers exist:

• T1 covers the primary registers of a field, such as legislation, official judgment feeds or citation indexes.
• T2 covers official government bodies and regulators.
• T3 covers peak bodies and professional associations.
• T4 covers trusted media and commentary.

Each row also carries one of three reuse constraints so that publisher terms are respected:

• REUSE_OK means the full text can be stored because an open licence permits it.
• LINK_SUMMARISE_ONLY means only a hyperlink and a short extract are kept.
• NO_BULK removes the endpoint from automated retrieval and sends it for manual review.

Rows additionally record what has been learned about the host, such as crawl delays to honor or a limit for hub pages, for instance fewer than three dated articles. Without those notes a row is a static bookmark that a collector cannot safely run.

The same register structure transfers to other fields:

• For scientific research, T1 rows point at citation indexes such as MEDLINE, Crossref and OpenAlex, combined with open-access full-text repositories and journal feeds configured to keep only abstracts or links.
• For managing personal health, rows point to official practice guidelines for a given condition and to medicine product information from the regulator, plus vetted clinical feeds.
• For legal and court watching, the main T1 row follows the court's official judgment feed.

A dynamic research lane complements the fixed list so that the base does not go stale. Each morning it runs a handful of targeted searches over a secondary list of origins, limited by word caps, a small number of capture attempts per source and a total page budget per run. For temporal accuracy, any page without a verifiable publication date, or with a future date, is dropped automatically. Open consultations with closing dates are the one exception: they sit on a watchlist, get fetched again every day and are marked as ongoing issues. Pages from hosts not on the list receive a low authority tier, so outside commentary is labeled as what it is.

Reading the top-tier registers directly

Every field has an apex source that outranks the rest, whether a legislative register, a primary citation index or a court feed. The collector reads these through structured APIs. The NDIS pipeline, for instance, gets primary legislation from the Federal Register of Legislation OData API. It prefers structured EPUB files to PDFs and runs a daily delta check against a persisted cursor to isolate what is new. Long legislative texts are cut into 60,000-character chunks and extracted piece by piece so that model limits are not hit.

Explicit identifiers separate a real legal change from a routine re-indexing:

• As-made instruments are freshly enacted law passed by Parliament, and they trigger active change notifications.
• Compilations are consolidated master texts with past amendments folded in. The ingestion planner marks them as background reference, so routine re-indexing never produces false change alerts.

Ingestion apart from publication

Raw artifacts are hashed with SHA-256 and written to Postgres before any synthesis begins, so the store grows on every run whether or not an edition goes out. Publication state is fixed deterministically with a UNIQUE(briefing_date, channel) constraint holding the serialized payload and its JSON hash. If a run dies or delivery to the channel fails, the next run finds the pending reservation and posts the frozen payload in order instead of writing new prose.

Three kinds of morning leave different traces:

• On a quiet morning with no new facts, documents are marked as assessed in one transaction and retired.
• On a withheld morning, where a critic rejected the draft, no assessment row is written and the documents return to the next morning's queue.
• On a published morning, the cited documents are marked processed and retired.

Persisting first gives strict temporal traceability. A question asked in March about what was known in January is answered strictly from passages kept in January, untouched by later edits to the live web.

The editorial desk

To prevent hallucination, drafting follows one hard rule: the model never handles ground-truth facts directly.

Code removes every literal date and URL from an evidence pack before it reaches the drafter. Deterministic pattern checks reject any draft that claims legal or operational status, such as in force or commenced. Verified dates return only after a draft is accepted, reattached from database fields so their provenance survives in the text. The rule helps in every domain mentioned: sample sizes, effect estimates and medication doses stay locked to database records and out of the model's parametric memory.

The desk follows a deterministic, multi-stage workflow:

1. Planning and exclusion. A planner assigns each document to a story or records an explicit reason for leaving it out, such as duplicate, immaterial or insufficient evidence.
2. Drafting and validation. The drafter writes a concise brief, capped at under 400 words with a firm limit on quoted excerpts. A deterministic validator checks every citation before a critic reads anything.
3. Review by two critics. The first critic can approve, reject or ask for one targeted revision. A stateless second critic then reviews independently, restricted to unresolved date or status questions. A rejection from either withholds the edition.
4. Rendering with provenance. Before anything is posted, the renderer pulls verified dates and links back out of the store and puts them into the text.

JSON schemas, deterministic validators, two critics and a renderer define the contract. Public text can be processed with inference hosted anywhere, but stores holding private health information or unpublished research must run inference inside local sovereign boundaries.

Querying through MCP, and joining private context per session

A read-only server implementing the Model Context Protocol exposes the persistent store through three tools: search_knowledge_base, get_document and list_changes. For factual integrity, search excerpts produced on the server side must end on word boundaries. A passage cut mid-word invites a generative model to complete the missing text plausibly in the presentation layer, a hazard that cannot be tolerated with medication doses or legal limits.

The public evidence is identical for everyone in a field, but each user arrives with private context. A compliance team wants a new rule weighed against internal policy, and a clinician wants updated research read against one patient's history. Mixing public evidence and sensitive local records in a single database would create serious data lineage and privacy liabilities.

The architecture therefore separates the public store from private context. Sensitive personal and clinical data stay within authenticated local environments behind a strict trust boundary. Verified public facts come from the read-only MCP server on request, and the assistant combines them with local records solely within the user's current session. Private context never enters the shared store, which keeps the public database shareable across many users.

Why open search and plain retrieval fall short

Pointing an autonomous agent straight at open web search, or at a standard RAG connector, does not meet the trust requirements of high-stakes fields. Live search ignores publisher reuse constraints, keeps no hashed provenance that an auditor could inspect, and cannot say what changed since yesterday. When questioned beyond its indexed scope, an ungrounded assistant tends to guess or finish truncated facts, and the errors flow into downstream analysis.

The same design maps onto several settings:

• For the NDIS and wider disability sector, changes to law and pricing give the public policy backdrop, and participant plan budgets and case notes meet it only during a live session.
• In scientific research, primary citation indexes and open-access repositories track the published literature while unpublished laboratory findings stay isolated.
• In health management, guidelines from clinical authorities and regulator data on medicines provide verified public evidence, and diagnostic and medical records remain private and on the local side.
• In legal and regulatory compliance, legislative registers and court judgment feeds set the public baseline, with internal policy manuals safely behind local authentication.

Leaving factual precision to an ungrounded language model is a gamble. A structured source register, a database that persists first, an editorial desk with two critics and a read-only MCP server underneath the assistant together turn fleeting web updates into a knowledge base that is authoritative, privacy-preserving and dependable.