Anchoring LLM Fine-Tuning With a Hand-Verified Real-Data Seed

Train on nothing but synthetic data, cycle after cycle, and the model drifts a little further from the real knowledge it was meant to absorb. By the third or fourth retrain it's good at imitating how domain text sounds and poor at stating what the domain says.

The remedy is a real-data seed: a small set of examples, checked by hand, taken from authoritative sources before any synthetic expansion starts. It has to be small enough that you can still verify every row. It has to be authoritative, because the model will treat the rows as ground truth. And it often has to come from places ordinary crawlers can't reach, since much of the most valuable domain knowledge sits behind access controls.

What follows covers collecting such a seed, shaping it into training rows, validating it, and checking whether the fine-tune held up. The topics, in order:

• why a purely synthetic loop drifts
• picking the right sources
• fetching protected pages as clean Markdown
• handling PDFs and other non-HTML documents
• turning the content into instruction-output pairs, preference pairs or raw domain text
• validating the seed and measuring the result

Why synthetic-only loops drift

Fine-tuning behaves statistically unlike pretraining. Pretraining spreads noise across billions of tokens, so a single bad paragraph vanishes into the average. A fine-tuning set might hold a few hundred rows, and there one mislabeled extraction makes up a measurable slice of the whole training signal.

In the usual synthetic pipeline, a generator writes examples using the teacher model's habits. A judge then filters them against those same habits, and the student absorbs both the habits and their biases. With no outside anchor, every retrain compounds the previous one. The model learns to sound like domain text rather than to reflect it.

A real seed interrupts that cycle. It gives the generator an anchor and gives the judge something to compare against that isn't the judge's own output. Size isn't the point. A few hundred strong examples usually suffice for one extraction task. What counts is that each row traces to a source you can vouch for, through an extraction you can check.

The catch is that authoritative material tends to be the hardest to retrieve. It hides behind JavaScript rendering, IP-based rate limits and access rules that stop standard crawlers.

Choosing what goes in

Decide on the contents before writing any code. For an ecommerce model, think product catalogs and verified customer reviews. A financial model calls for SEC filings, earnings transcripts and regulatory guidance. A medical one needs clinical practice guidelines, drug labels and peer-reviewed studies.

Four criteria apply across domains:

• Authoritative: practitioners in the field trust it, not just a search engine.
• Structured: the content follows consistent patterns that an extraction schema can aim at.
• Current: it describes the domain as it stands today, not as it stood two years ago.
• Relevant: it ties directly to the task instead of sitting next to it.

Leave out SEO content farms, AI-generated text, and anything whose origin you couldn't name if someone asked where the training data came from.

Fetching protected pages

A bare requests.get() call sends a plain TLS fingerprint and no browser headers, and protected sites spot that right away. If the target renders with JavaScript, the response is an empty shell.

A fetch API can do the rendering, the access work and the Markdown conversion in a single request. In the example, mode=auto switches on adaptive stealth: the service alternates JavaScript rendering with proxy escalation for each request and settles on whatever configuration the target needs.

import os
import requests
from dotenv import load_dotenv

load_dotenv()

apikey = os.getenv("ZENROWS_API_KEY")
url = "https://www.scrapingcourse.com/ecommerce/"

params = {
    "url": url,
    "apikey": apikey,
    "mode": "auto",
    "response_type": "markdown",
}

response = requests.get("https://api.zenrows.com/v1/", params=params)
print(response.text)

The payload that comes back is smaller and cleaner than raw HTML, since most navigation, footer text and boilerplate are already gone. Less noise on the input side means less noise for the model to learn from.

PDFs and other non-HTML sources

Many of the best domain sources aren't web pages. Drug labels, regulatory filings, clinical guidance and technical standards are all distributed as PDFs. For those, set response_type=pdf, and also js_render=true, which PDF responses require.

params = {
    "url": url,
    "apikey": apikey,
    "js_render": "true",
    "response_type": "pdf",
}

response = requests.get("https://api.zenrows.com/v1/", params=params)

with open("source-document.pdf", "wb") as f:
    f.write(response.content)

The service delivers the file but doesn't extract the text inside it. Pull that out with pypdf or pdfplumber, then split it into chunks before any of it becomes a training example. Cut along structure instead of a fixed character count. A document's own headings and sections are natural boundaries, so each chunk keeps its context and no argument gets split in the middle.

Recording provenance

An authoritative seed only stays authoritative if you can say where each row came from. Attach a metadata field holding the source URL, the retrieval date and the license, and the dataset remains auditable and defensible.

source = {
    "url": pdf_url,
    "retrieved": date.today().isoformat(),
    "license": license_url,
}

This matters most when someone questions the model's outputs. You'll want to state which row came from which source, and whether that source permits this use.

Three shapes for a training row

A fine-tuning example can take three forms, and each teaches something different.

Instruction-output pairs are the usual choice for supervised fine-tuning. Every example couples the Markdown you fetched from a page with the right structured result, so the model picks up both what to extract and the format to return it in.

def to_training_example(instruction, markdown_content, extracted_json):
    return {
        "messages": [
            {"role": "system", "content": instruction},
            {"role": "user", "content": markdown_content},
            {"role": "assistant", "content": json.dumps(extracted_json)},
        ]
    }

Preference pairs serve DPO and RLHF. They hold two outputs for the same input, with one marked as preferred. Use them to nudge a model that already produces decent results toward better ones.

def to_preference_pair(prompt, chosen, rejected):
    return {
        "prompt": prompt,
        "chosen": chosen,
        "rejected": rejected,
    }

Domain text continuation is plain domain text with no input-output split. It familiarizes the model with vocabulary and style, but it's weak for extraction, because the model never sees what should be pulled out.

For extraction work, then, instruction-output pairs fit best. The other two shapes serve other goals.

Filtering before anything enters the seed

Not every scraped page deserves a place in the dataset. Each row has to pass three checks first:

def keep(markdown, extracted):
    if len(markdown.strip()) < 200:
        return False
    try:
        validate(instance=extracted, schema=schema)
    except ValidationError:
        return False
    h = hashlib.sha256(" ".join(markdown.lower().split()).encode()).hexdigest()
    if h in seen:
        return False
    seen.add(h)
    return True

The first test drops pages that came out nearly blank after rendering. The second, a schema check, rejects wrong types and empty fields. The third compares content hashes, which flags exact copies and copies differing only in formatting. A page must pass every test to reach training.

Measuring whether it worked

Before synthetic data enters the picture, carve off 10 to 20 percent of the real seed and never train on it. When fine-tuning finishes, run the model over that holdout and count the outputs that match the expected answer.

random.seed(42)
random.shuffle(seed_examples)

split = int(len(seed_examples) * 0.85)
train_examples = seed_examples[:split]
holdout_examples = seed_examples[split:]

Repeat the check after every retrain and log the score over time. A steady or rising number says the seed is working. A falling one is an early warning of drift, visible before it compounds.

Points to remember

• A seed of a few hundred high-quality real examples is worth more than a large synthetic dataset that lacks one.
• Authoritative sources nearly always sit behind access controls that standard crawlers can't get past.
• Adaptive mode deals with JavaScript rendering and bot detection per request, with no per-site setup.
• Chunk PDFs by structure rather than character count so each piece stays coherent.
• Screen each row on the way in, since a single bad extraction weighs a lot in a small training set.
• Reserve a slice of real data ahead of any synthetic growth, then score the tuned model on it.

For collecting at larger scale, a batch endpoint that handles concurrency and retries over long URL lists is the natural next step.