Fine-Tuning a 7B Medical Model for Free: QLoRA, FAISS and Safety Guardrails

Medical chatbots tend to fail in one of two ways. Either they invent treatments, or they blend traditional and evidence-based medicine into one dangerous stew. The goal here was a model that keeps the two worlds strictly apart, shows where its information comes from and steps aside when someone describes an emergency. It's called Expert Vaidya, it separates Ayurvedic medicine from modern clinical guidelines, and the whole thing was built at no cost.

Hardware was two Tesla T4 GPUs on Kaggle's free tier. The model is a 7B open-weights one, and the knowledge comes from public-domain classical texts. The result runs offline on a 16 GB laptop and installs with one command. Here's the full pipeline.

The pieces

• Base model: Qwen2.5-7B-Instruct, tuned using Unsloth and QLoRA
• Training data: 52 structured examples, 2 written by hand and 50 synthetic ones that passed a quality filter
• Retrieval: a FAISS index over 6,765 chunks pulled from 1,492 pages of the Sushruta Samhita (public domain on Archive.org), plus a verified knowledge base of 22 entries drawn from WHO, ADA, NICE and PubMed Central
• Safety: guardrail code in plain Python, with no model involved, that screens each question ahead of the LLM
• Delivery: a GGUF Q4_K_M file (4.68 GB) published to Hugging Face and the Ollama registry

Step 1: cheap synthetic data

Labeled medical data costs more than a hobby budget allows, so a smaller model, Qwen2.5-3B in 4-bit, generated the examples across 50 medical topics under a strict output contract. Each answer had to include an Ayurvedic section, a modern section and a historical section, and anything incomplete was thrown out automatically:

python has_all_3 = all(x in response for x in ["Ayurvedic", "Modern", "Historical"])
if has_all_3 and len(response) > 200:
    keep(example)

All 50 generated examples passed. With the 2 handwritten seeds that makes 52 training rows. It's a tiny set, but the aim wasn't to teach facts. It was to teach format and discipline. Facts come from retrieval.

Step 2: QLoRA on free T4s

Unsloth on Kaggle's two T4s trained only LoRA adapters:

    model = FastLanguageModel.get_peft_model(
    model, r=16,
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                    "gate_proj","up_proj","down_proj"],
    lora_alpha=16, lora_dropout=0,
    use_gradient_checkpointing="unsloth",
)

Only 0.53% of the weights moved: 40,370,176 trainable parameters among 7,655,986,688 overall, 5 epochs, 35 steps, an effective batch size of 8 and a final training loss of 1.0493. Wall time was under 10 minutes. No medicine was learned here. The model learned to always reply in the five-part structure and never mix the domains.

Step 3: 1,492 pages into FAISS

Format doesn't ground answers by itself. Two public-domain English translations of the Sushruta Samhita (812 and 680 pages) came from Archive.org. PyMuPDF extracted the text, LangChain's recursive splitter cut it into chunks of 500 characters with 100 overlap, and all-MiniLM-L6-v2 embedded them. That produced 6,765 chunks, normalized and loaded into a FAISS IndexFlatIP.

At query time the system pulls the top-k chunks but applies a relevance floor, a score of at least 0.3 to 0.4, so irrelevant pages never reach the context window.

Step 4: a verified layer

Old translations scanned from print are noisy OCR, so a second, curated index sits beside the first. It holds 22 entries checked by hand, each with source attribution and a confidence score. Sources include the Charaka Samhita, Sushruta Samhita and Ashtanga Hridaya, WHO guidelines from 2023 and 2024, the ADA Standards of Care 2025, NICE red-flag guidance (NG185 and NG173), plus three PubMed Central trials: Gymnema PMC2915138, Boswellia PMC3251738 and Ashwagandha PMC3573577. A table of herb and drug interactions for warfarin and diabetes medication rounds it out. Retrieval keeps only entries with verified == True and confidence >= 0.8.

Step 5: guardrails run first

The key design decision is that emergencies never reach the model. Plain Python inspects the input before anything else:

warnings, disclaimer = safety_guard.generate_safety_response(user_question)
if warnings and "EMERGENCY" in warnings[0]:
    return warnings[0] + disclaimer   # LLM never runs

It watches 14 emergency indicators, such as chest pain, stroke signs, anaphylaxis and suicidal statements, plus 7 dangerous self-treatment patterns like "stop taking," "replace insulin" or "double the dose." A chest-pain message returns a triage warning with emergency numbers (112, 911, 999), the same way every single time.

Step 6: quantize, publish, deploy

After merging the LoRA weights, Unsloth wrote out a 4.68 GB GGUF in the Q4_K_M format, which went up to Hugging Face and the Ollama registry. Anyone can run the model now:

bash   ollama run tejaas6/expert-vaidya

The system prompt, temperature and guardrail behavior ship inside the model. You need no setup and no API key, and it works fully offline.

What went wrong

Here's the part few people write up:

• An endless listing loop. At low temperature the model once produced hundreds of variants of "hyaluronan polymer...". Setting no_repeat_ngram_size=3 stopped it.
• A Hugging Face upload crash. A 4.4 GB file triggered an XetProgressReporter ImportError. The fix was setting HF_HUB_DISABLE_XET=1 to skip the xet transfer path and clearing cached modules before importing again.
• Kaggle's disappearing /tmp. Unsloth sent the GGUF export to /tmp because the working directory was short on space, and /tmp isn't kept as kernel output. An explicit shutil.copy into /kaggle/working before zipping solved it.
• Language leakage. Qwen sometimes slipped Chinese characters into answers. A regex post-processor and an English-only rule in the prompt cleaned that up.

Results

• A diabetes question cites the Charaka Prameha classification and the WHO and ADA diagnostic thresholds, each kept separate.
• Asking whether Gymnema can be taken alongside diabetes drugs flags the additive hypoglycemia risk and the monitoring protocol.
• Describing crushing chest pain skips the LLM entirely and returns pure triage advice.
• A made-up disease gets a refusal, not an invented treatment.

Where to find it

You can pull the model from the Ollama registry, the GGUF weights are on Hugging Face under Expert-Vaidya-Qwen7B-GGUF. This is an educational research project. It isn't medical advice and it isn't a diagnostic tool.