Domain-aware RAG with MLflow registries and LLM routing

Most RAG demos wire one embedder, one collection, and one LLM. That is fine until mortgage questions and medical questions share a process, or until a small local model would suffice for triage while a larger remote model should answer clinical detail. A sturdier pattern classifies intent first, selects the matching knowledge base and embedding model, retrieves context, then routes generation to an appropriate LLM.

This build runs that idea on modest hardware: MLflow Model Registry on a Raspberry Pi for embedding champions, Qdrant collections per domain, Ollama for local classification, and selective cloud LLM calls for the domains that need them. The networking pain of pointing the Pi at Postgres is part of the story, because model registries are only useful if the backend store is reachable.

Postgres connectivity from the Pi

If Postgres was installed via Homebrew on a Mac on the same LAN, confirm which config the running instance uses before editing:

psql -U <your_user> -d postgres -c "SHOW config_file;"

Listen addresses and related settings live in postgresql.conf:

cd /opt/homebrew/var/postgresql@15
sed -i '' "s/#listen_addresses = 'localhost'/listen_addresses = '*'/" postgresql.conf

Verify the effective configuration:

grep -v "^#" postgresql.conf | grep listen_addresses
# listen_addresses = '*'

Whitelist the Pi’s IP in pg_hba.conf so password auth is allowed from that host:

echo "host  all  all  192.168.1.9/32  md5" >> pg_hba.conf

Verify the rules loaded as you expect:

grep "192.168.1.9" pg_hba.conf

Restart Postgres and confirm it listens on the interfaces you intend:

brew services restart postgresql@15
lsof -i -P | grep LISTEN | grep postgres

Confirm the role exists with login privileges:

psql -U <admin_user> -d postgres -c "\du"

Set or reset the password MLflow will use:

psql -U <admin_user> -d postgres -c "ALTER USER root WITH PASSWORD 'root';"

Ensure the target database exists:

psql -U <admin_user> -d postgres -c "\l" | grep mlflow
psql -U <admin_user> -d postgres -c "CREATE DATABASE mlflow OWNER root;"

Test raw connectivity from the Pi before involving MLflow:

nc -zv 192.168.1.8 5432

When the network path and credentials work, start MLflow against that remote backend store:

mlflow server \
  --backend-store-uri postgresql://root:root@192.168.1.8:5432/mlflow \
  --host 0.0.0.0 \
  --port 5000 \
  --allowed-hosts "192.168.1.9:5000" \
  --cors-allowed-origins "http://192.168.1.9:5000"

Architecture in two stages

Stage 1 — knowledge layer. Domain PDFs are chunked and embedded with domain-specific sentence-transformer models, then stored in separate Qdrant collections. MLflow tracks which embedder build is champion for each domain so services do not hard-code local paths.

Stage 2 — intelligent routing. A classifier labels each user question (for this demo: leukemia, mortgage, or other). Retrieval uses the matching collection and embedder. Generation uses the LLM chosen for that intent—local for some paths, remote for others—so cost and capability track the question.

Implementation modules

model_registry.py registers Arctic-embed for mortgage and MiniLM for leukemia (or your chosen pair), logs them to MLflow on the Pi, and aliases champions:

import mlflow
import mlflow.sentence_transformers
from mlflow import MlflowClient
from mlflow.models import infer_signature
from sentence_transformers import SentenceTransformer

mlflow.set_tracking_uri("http://192.168.1.9:5000")
mlflow.set_experiment("embedding-models")

EMBEDDERS = {
    "mortgage-embedder": {"source": "Snowflake/snowflake-arctic-embed-m-v1.5"},
    "leukemia-embedder": {"source": "all-MiniLM-L6-v2"},
}

sample_texts = ["MLflow tracks embedding models", "Qdrant stores vectors"]
client = MlflowClient()

for name, cfg in EMBEDDERS.items():
    model = SentenceTransformer(
        cfg["source"],
        trust_remote_code=cfg.get("trust_remote_code", False),
        config_kwargs=cfg.get("config_kwargs"),
        device=cfg.get("device"),
    )
    with mlflow.start_run(run_name=name):
        info = mlflow.sentence_transformers.log_model(
            model=model,
            name="embedder",
            signature=infer_signature(sample_texts, model.encode(sample_texts)),
            input_example=sample_texts,
            registered_model_name=name,
        )
    client.set_registered_model_alias(name, "champion", info.registered_model_version)
    print(f"{name}: v{info.registered_model_version}, dim={model.get_embedding_dimension()}")

ingest_data.py rebuilds domain collections, chunks PDFs with a consistent token window and overlap, embeds with the champion model for that domain, and upserts into Qdrant:

import os
from chonkie import Pipeline
from qdrant_client import QdrantClient
from dotenv import load_dotenv, find_dotenv

load_dotenv(find_dotenv())

client = QdrantClient(url="http://localhost:6333", api_key=os.environ["QDRANT_API_KEY"])
client.delete_collection("leukemia_details") # this will delete and recreate the collection.

(Pipeline()
 .fetch_from("file", dir="../", ext=[".pdf"])
 .process_with("liteparse")
 .chunk_with("recursive", chunk_size=512)
 .refine_with("overlap", context_size=100)
 .store_in("qdrant", collection_name="leukemia_details", client=client,
           embedding_model="sentence-transformers/all-MiniLM-L6-v2")
 .run())

count = client.count("leukemia_details", exact=True).count
print(f"Ingested {count} chunks into leukemia_details")

ai_gateway.py classifies intent with a strict local prompt via Ollama, then chooses LLM backends per label:

from openai import OpenAI
import mlflow

mlflow.set_tracking_uri("http://192.168.1.9:5000")

client = OpenAI(
    base_url="http://192.168.1.9:5000/gateway/mlflow/v1",
    api_key="no-api-key",
)

CLASSIFIER_PROMPT = (
    "Classify the user query into exactly one label: LEUKEMIA, MORTGAGE, or OTHER.\n"
    "LEUKEMIA = questions about leukemia (AML, CML, ALL, CLL), its symptoms, diagnosis, "
    "treatment, prognosis, or related blood cancers.\n"
    "MORTGAGE = questions about home loans, mortgage rates, refinancing, EMI, "
    "eligibility, lenders, or mortgage documents.\n"
    "OTHER = everything else, including general medical questions not about leukemia.\n"
    "Reply with only the label."
)

ROUTES = {
    "LEUKEMIA": "openai",
    "MORTGAGE": "gemma",
}

REJECT_MESSAGE = (
    "Sorry, this setup can only answer Leukemia-specific medical questions "
    "and mortgage-related questions."
)


def classify(query: str) -> str:
    resp = client.chat.completions.create(
        model="ollama",  # local Ollama endpoint does the classification
        messages=[
            {"role": "system", "content": CLASSIFIER_PROMPT},
            {"role": "user", "content": query},
        ],
        temperature=0,
    )
    label = resp.choices[0].message.content.strip().upper()
    for intent in ROUTES:
        if intent in label:
            return intent
    return "OTHER"


def route(query: str) -> str:
    intent = classify(query)
    target = ROUTES.get(intent)
    if target is None:
        print(f"[intent={intent} -> rejected]")
        return REJECT_MESSAGE
    resp = client.chat.completions.create(
        model=target,
        messages=[{"role": "user", "content": query}],
    )
    print(f"[intent={intent} -> model={target}]")
    return resp.choices[0].message.content

#
# if __name__ == "__main__":
#     print(route("What are the first-line treatments for acute myeloid leukemia?"))
#     print(route("How does refinancing affect my monthly mortgage EMI?"))
#     print(route("What are the side effects of metformin?"))   # medical but not leukemia -> rejected
#     print(route("Write a haiku about monsoon in Hyderabad."))  # rejected

smart_rag.py ties classification, collection selection, top-k retrieval, and grounded answering together:

from qdrant_client import QdrantClient
import os

from ai_gateway import client, classify, ROUTES, REJECT_MESSAGE
from embedding_model_inference import EMBEDDERS
from dotenv import load_dotenv, find_dotenv

load_dotenv(find_dotenv())

COLLECTIONS = {
    "LEUKEMIA": {"collection": "leukemia_details", "embedder": "leukemia-embedder"},
    "MORTGAGE": {"collection": "mortgage", "embedder": "mortgage-embedder"},
}
TEXT_FIELD = "text"
EXIT_WORDS = {"bye", "exit", "quit"}


class SmartRAG:
    def __init__(self, qdrant_url="http://localhost:6333", top_k=5):
        self.qdrant = QdrantClient(url=qdrant_url, api_key=os.environ.get("QDRANT_API_KEY"))
        self.top_k = top_k

    def retrieve(self, query, intent):
        cfg = COLLECTIONS[intent]
        embedder = EMBEDDERS[cfg["embedder"]]
        prompt = "query" if "query" in embedder.prompts else None  # arctic needs its query prefix
        vec = embedder.encode(query, prompt_name=prompt, normalize_embeddings=True).tolist()
        hits = self.qdrant.query_points(cfg["collection"], query=vec, limit=self.top_k, with_payload=True).points
        return "\n\n".join(str(h.payload.get(TEXT_FIELD, h.payload)) for h in hits)

    def answer(self, query):
        intent = classify(query)
        if intent not in ROUTES:
            return REJECT_MESSAGE
        context = self.retrieve(query, intent)
        resp = client.chat.completions.create(
            model=ROUTES[intent],
            messages=[
                {"role": "system", "content": f"Answer using only this context:\n{context}"},
                {"role": "user", "content": query},
            ],
        )
        print(f"[intent={intent} -> {COLLECTIONS[intent]['collection']} -> {ROUTES[intent]}]")
        return resp.choices[0].message.content

    def run(self):
        while True:
            query = input("\nyou> ").strip()
            if query.lower() in EXIT_WORDS:
                break
            if query:
                print("\nassistant>", self.answer(query))


if __name__ == "__main__":
    SmartRAG().run()

embedding_model_inference.py loads embedders from MLflow by alias (models:/...@champion) so online paths never depend on a developer laptop path:

import mlflow
import mlflow.sentence_transformers

mlflow.set_tracking_uri("http://192.168.1.9:5000")

EMBEDDERS = {
    name: mlflow.sentence_transformers.load_model(f"models:/{name}@champion")
    for name in ("leukemia-embedder", "mortgage-embedder")
}

QUERIES = {
    "leukemia-embedder": "What are the first-line treatments for acute myeloid leukemia?",
    "mortgage-embedder": "How does refinancing affect my monthly mortgage EMI?",
}

for name, embedder in EMBEDDERS.items():
    vecs = embedder.encode([QUERIES[name]], normalize_embeddings=True)
    print(f"{name}: shape={vecs.shape}, prompts={list(embedder.prompts) or None}")

Results

A mortgage question should retrieve from the mortgage collection with the mortgage embedder and answer under the mortgage LLM policy. A leukemia question should follow the clinical path. Off-domain questions should abstain or answer without pretending domain retrieval succeeded.

/Users/pavanmantha/.local/bin/uv run /Users/pavanmantha/Pavans/PracticeExamples/DataScience_Practice/mlflow-tutorials/mlflow_on_edge/.venv/bin/python /Users/pavanmantha/Pavans/PracticeExamples/DataScience_Practice/mlflow-tutorials/mlflow_on_edge/src/smart_rag.py
Downloading artifacts: 100%|██████████| 16/16 [00:19<00:00,  1.19s/it]
2026/09/06 12:48:35 INFO mlflow.sentence_transformers: 'models:/leukemia-embedder@champion' resolved as 'mlflow-artifacts:/3/models/m-f568ee233dd748f997ba66763e41653b/artifacts'
Downloading artifacts: 100%|██████████| 1/1 [00:00<00:00, 76.69it/s]
Loading weights: 100%|██████████| 103/103 [00:00<00:00, 15587.15it/s]
Downloading artifacts: 100%|██████████| 16/16 [01:29<00:00,  5.60s/it]
2026/09/06 12:50:06 INFO mlflow.sentence_transformers: 'models:/mortgage-embedder@champion' resolved as 'mlflow-artifacts:/3/models/m-c4221c844dfb45bfa15c6a4ecde56920/artifacts'
Downloading artifacts: 100%|██████████| 1/1 [00:00<00:00, 27.17it/s]
Loading weights: 100%|██████████| 197/197 [00:00<00:00, 8855.57it/s]
[transformers] BertModel LOAD REPORT from: /var/folders/1x/36nz44_569501n4xs0px2_8m0000gn/T/tmpef4y6tj5/model.sentence_transformer
Key                 | Status  |
--------------------+---------+-
pooler.dense.weight | MISSING |
pooler.dense.bias   | MISSING |

Notes:
- MISSING: those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.
leukemia-embedder: shape=(1, 384), prompts=['query', 'document']
mortgage-embedder: shape=(1, 768), prompts=['query', 'document']
/Users/pavanmantha/Pavans/PracticeExamples/DataScience_Practice/mlflow-tutorials/mlflow_on_edge/src/smart_rag.py:20: UserWarning: Api key is used with an insecure connection.
  self.qdrant = QdrantClient(url=qdrant_url, api_key=os.environ.get("QDRANT_API_KEY"))

you> What are the phases of treatment for acute lymphoblastic leukemia?
[intent=LEUKEMIA -> leukemia_details -> openai]

assistant> The phases of treatment for acute lymphoblastic leukemia (ALL) are:

1. Induction - achieving remission.
2. Consolidation - eradicating residual disease with different agents to circumvent resistance.
3. Maintenance (in ALL) - prolonged low-dose therapy to suppress re-emergence of the clone.

you> What was 30-day Average SOFR referred to in Series issued before July 2023?
[intent=MORTGAGE -> mortgage -> gemma]

assistant> *   Question: "What was 30-day Average SOFR referred to in Series issued before July 2023?"
    *   Constraint: Use *only* the provided context.

    *   Chunk 1: Discusses "Excess inclusion income", Residual Holders, taxable income, Trust REMIC taxable income, etc. No mention of SOFR or July 2023.
    *   Chunk 2: Discusses "Backup withholding", Non-U.S. Persons, Ginnie Mae Guaranty, State/Local/Foreign tax. No mention of SOFR or July 2023.
    *   Chunk 3: Discusses "Residual Holder's share of expenses", REITs, RICs, phantom income. No mention of SOFR or July 2023.
    *   Chunk 4: Discusses "Principal Distribution Amount", Ginnie Mae Project Loan Certificates, Underlying Certificates. No mention of SOFR or July 2023.
    *   Chunk 5: Discusses "OID on a Teaser Security", "Initial Distribution Period", "Excess Interest Payment", "Accrual of Original Issue Discount". No mention of SOFR or July 2023.

    *   I have scanned all five provided text nodes.

you>

Why this shape is interesting

Registries make embedders swappable without redeploying every client. Per-domain collections avoid cross-domain nearest-neighbor pollution. Intent routing keeps expensive models off trivial traffic. Running MLflow on a Pi is not required for the pattern, but it proves the registry can live on controlled hardware close to the lab network.

Operational cautions

Secure Postgres before exposing it on a LAN. Treat pg_hba changes as production changes. Pin MLflow and driver versions. Monitor classifier error rates: a wrong intent sends the query to the wrong collection and can look like a “bad LLM.” Evaluate retrieval and routing separately. When champions flip in the registry, run a smoke ingest or a shadow embed comparison before cutting live traffic.

Closing

RAG beyond a single retrieval path is mostly product architecture: know the domain, fetch with the right geometry, generate with the right budget. MLflow, Qdrant, and a small gateway make that architecture concrete enough to run on a desk—and disciplined enough to grow past a demo.

Hardening the registry-backed path

Champion aliases should move through staging. Log the model version ID serving each answer so you can replay incidents after a registry change. If the Pi goes offline, clients need a fail-closed mode rather than silently falling back to a random local checkpoint.

Chunking parameters belong in the registry metadata next to the embedder. A new champion with a different preferred context length should not reuse old chunks without a rebuild flag.

Classifier prompts deserve fixtures: ten mortgage phrasings, ten clinical phrasings, ten adversarial “other” phrasings. Measure precision/recall before you tune LLM choice. Otherwise you will chase generator quality while routing is the true defect.

For Raspberry Pi deployments, watch disk wear from frequent artifact writes and prefer external disks for MLflow artifact roots. Keep Postgres backups on a different machine than the Pi.

When explaining the design to stakeholders, draw the sequence: classify → select embedder/collection → retrieve → select LLM → answer with citations. That picture sells better than a tool list and makes clear where policy decisions live.

Extend domains carefully. Each new label needs a collection, an embedder champion, ingestion job, classifier examples, and an LLM policy. The gateway should reject unknown labels rather than defaulting to the largest model.

Finally, keep the “OTHER” path honest. Refusing is better than retrieving from a random domain to avoid awkward silence. Users trust systems that know when they are out of scope. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further. Route accuracy and retrieval quality must be measured as separate release gates before generator prompts are tuned further.