Agentic Finetuning: Training a Knowledge Wiki Instead of Model Weights

Knowledge a company owns and can't reach

Imagine a machine maker holding about 900 service reports. The pile hides answers to questions worth real money: which machine series breaks down most and for what reason, which repairs hold up, and what fails in winter installs yet never in summer ones.

Here's the awkward part: nobody can hand over those answers. A few veteran employees carry pieces in memory, yet no document spells anything out in full. Reading all 900 reports is beyond any single person. So the knowledge sits inside the company and still can't be reached by it. Nearly every organization beyond a certain age lives with this, and most AI projects today leave it alone.

What companies really want

Companies don't want to search their data, since Ctrl+F settled search long ago. They want the learnings. Compare the lookup versions of questions with the learning versions:

• A lookup asks what test protocol 214 says. A learning question asks how a concrete mix behaves in cold pours across fifteen years of test reports.
• A lookup asks for the patent document from one particular case. A learning question asks how the filing process really goes, which objections recur and what gets an application rejected on the first try.
• A lookup asks for ticket 8812. A learning question asks what the most experienced support staff do when a machine fails twice in a month, and what they promise the customer.

The learnings carry the value, and they behave cruelly: they exist only as fragments and never sit in one spot. Nothing on paper holds them. They emerge between documents, as patterns spanning hundreds of reports, as the process that crystallizes out of three hundred cases, as a rule two senior people keep in their heads. If those two leave, what walks out the door is the company's memory, not merely two staff members.

Why "chat with your documents" falls short

The usual industry recipe is to embed everything, fetch the closest chunks and generate text from them. For lookups that works well, and this isn't an attack on RAG. Still, retrieval hands back what somebody wrote down. Pose a learning question, such as which series fails most and why, and it pulls three chunks from three reports, then improvises a confident-sounding synthesis on the spot, and a different one next time. Nobody extracted the learning, checked it or gave it a stable form. Instead of querying knowledge, the system reconstructs it per question, and poorly.

The pile itself is also a mess, with contradictory policy versions, outdated specs and mistakes copied forward. Any system reading the raw pile inherits those flaws. Imagine two PDFs, one promising 12 months of warranty and the other 24. Both get retrieved, and the answer becomes a coin flip.

Even a one-time human synthesis, whether a consultant's report or an intern's summary wiki, runs into two killer questions. How does anyone know it's right, and how does anyone know it's complete? There's no ground truth, no test and no definition of done, so the document starts rotting quietly the day it's finished.

So the true problem lies elsewhere than retrieval. It's converting vast amounts of data into learnings people can rely on, and keeping those learnings correct over time. Compressed to a phrase: mining diamonds out of a coal mine. They're in there, and hand-sifting the coal appeals to no one.

Why finetuning the model misses

Finetuning is a natural instinct, and it feels right for a reason. It's genuine conventional machine learning: data, labels, a loss, weight updates and a test set the model never saw. That's exactly the rigor the document-chat world lacks.

Yet it changes the wrong layer. Weights are opaque. When the cold-pour answer is wrong, there's no opening neuron four million to repair it. Whatever was learned freezes at training time and ages with the next wave of reports, and every refresh costs another training run, evaluation and deployment. Finetuning also has a reputation for teaching skills and style while storing facts unreliably. In short, the correct loop is wrapped around the wrong object.

Running the loop one layer up

The proposal is to keep the whole conventional ML training loop and run it one level higher, on the knowledge instead of the weights. The name for this is Agentic Finetuning.

The mapping is structural and not poetic: each part of the standard training loop has a direct counterpart, and the thing being trained is the knowledge.

The trainable artifact is a wiki, a set of readable pages of extracted learnings with every claim tied to its sources. One page might cover cold-pour behavior and another how filings actually go. A human can open such pages and argue with them. The agent on top stays deliberately plain and reads the wiki and nothing else, so all the intelligence lives in what the wiki holds. A wrong answer is fixed by repairing the knowledge, the way backprop repairs weights, and never by tweaking a prompt and hoping.

Read the diagram as one pass through the loop: deep mining first, then a benchmark drawn from the raw data, tuning until scores settle, and finally the secret questions, which no earlier step ever touches.

Step 1: dump plus purpose

Everything gets uploaded unsorted, together with an answer to an underrated question: what should be learned from this? Examples are how materials behave, or how filings actually go. The stated purpose pushes the mining toward learnings and away from summaries.

Step 2: deep mining, wiki v0

Next the full pile gets mined with care, document after document and region after region, to whatever depth the purpose demands. It's the costly stage and happens one time. Depth belongs up front for a reason. Suppose it showed up only where the benchmark later objected. The wiki would then take its shape from the exam, and the knowledge would have been tuned for the test and not for the data. Mining deep first lets the pile itself determine completeness, so the benchmark's role shrinks to checking and repair.

Contradictions found along the way, like the 12-versus-24-month warranty pair, go into a queue for a human. They don't block mining, and they're settled at the end, since no model knows which PDF is the outdated one.

Step 3: a benchmark built from the data, never from the wiki

People tend to stumble here first. If test questions come out of the wiki, anything missing from the wiki has no question probing it, so the blind spot sets its own exam and aces it. Questions therefore need three other sources: the raw documents (what ought a reader to take from this report?), the declared purpose (cross-document questions on trends, causes and processes) and the people involved (what they genuinely get asked). The set is then divided, 75 percent for training and 25 percent sealed away as secret questions with the same topic balance, no paraphrased overlap and no one allowed to view them. Teaching to the test would drain it of meaning.

Step 4: answer, grade, diagnose, rewrite

Every miss receives a root cause, and each cause has its own route:

• If the learning exists in the data and the wiki overlooked it, mine that region again, more deeply.
• If it exists nowhere, ask a human and never invent an answer.
• If two sources disagree, surface the conflict.
• If the question itself was flawed, repair the question and leave the agent alone.

One rule beats all the rest: the answer to a question is never copied into the wiki. The benchmark points to where digging is needed, the documents decide what is recorded, and what's recorded must be general pages that any later question can use. If that rule breaks, the wiki turns into an answer key without anyone noticing.

Step 5: the honesty check

When training scores stop climbing, the secret questions are run a single time. If training and secret results sit close together, the wiki really does hold the learnings, and it can be certified, versioned and shipped with its limits printed on the box. If they're far apart, the thing built is a parrot. The exam then gets renewed: burned questions are retired, new ones are drafted from the data pool with fresh wording and samples, a new sealed set is locked away and tuning resumes. Because the wiki had bent toward the old questions, later rounds face targets it hasn't seen. A drop from 95 on training to 71 on secret questions hurts, though the same overfit would have shown up in any Kaggle notebook in 2019. Old lessons carry over, which is the entire idea.

Production then supplies an endless test set. A question the agent mishandles becomes both a wiki fix and a new benchmark entry, and fresh reports restart the loop. In the end the company owns a living, tested memory that outlasts retirements.

The whole method fits in one sentence:

Benchmarks say where to dig, data says what to write, and the secret set says when to stop.

Where it fits

The method fits organizations whose value sits in finite piles they own. Engineering and materials companies with decades of reports and test protocols qualify, as do law and IP practices, where the real process lives across hundreds of cases. Manufacturers with manuals and service histories fit too, along with support teams holding years of conversations, whose benchmark is practically waiting in the export.

Buyers who demand proof benefit as well, since a certificate with secret-question scores carries more weight than a claim that the product runs on the latest model.

Two side effects become products themselves. A cleaned wiki makes superb onboarding material, because newcomers can read the learnings directly. And the audit finding, that two policies contradict each other while staff improvise, is what consultants charge companies a lot to discover. The loop yields it along the way.

Limits

The method earns trust by knowing its limits, so here they are:

• Skills, style and voice are out of scope. If the model can't reason or write the way a task needs, that's real model-finetuning territory. Agentic Finetuning tunes what the agent knows and not what it can do.
• World knowledge is out of scope. It masters one organization's pile and won't make the agent smarter about concrete in general, only about that company's concrete.
• Hourly-changing values don't belong. Live prices need a live lookup and not a wiki page. Bake them in and the result is a beautifully certified lie.
• Unarbitrated knowledge stays unresolved. Contradictions get surfaced by the loop, and someone who knows the truth must resolve them. When an organization truly can't say whether its warranty runs 12 or 24 months, no technique settles that, although an audit that records the dispute has worth by itself.
• Tiny piles don't justify it. With twenty documents, writing the wiki by hand takes an afternoon. The loop pays for itself when no human can hold the pile in mind.
• Taking actions is a different problem. Answering from knowledge is what this method does. Issuing refunds is another safety problem entirely, and anyone selling both in one breath deserves suspicion.

Boring parts, real edge

None of this is a research breakthrough. Test sets are old, LLM graders are routine and knowledge extraction is an established field. The edge is that hardly anyone does the dull parts. Teams compare models while the learnings that actually matter sit unread in a folder named FINAL_v2_old_DO_NOT_DELETE.

Models keep improving by themselves, but the lessons in a company's data won't extract themselves. The part worth training is the part that's actually the company's own.