From Base Model to Chat Model: SFT Tuning a 102M Indonesian LLM

After pretraining NovAI Base 102M, the next move wasn't scaling it up. It was teaching it to follow instructions. The base model is a 102-million-parameter decoder-only Transformer trained from scratch on roughly 2 billion Indonesian tokens, and it works: it writes Indonesian, continues a prompt and has picked up enough statistical structure to produce coherent text. But it isn't really a chatbot. Hand it a question and there's no guarantee of an answer, because it was trained to predict the next token and not to recognize this conversational pattern:

User → question
Assistant → answer

Larger GPUs weren't available for a bigger model, so the experiment kept the same model and changed the training objective and data instead. The method was Supervised Fine-Tuning (SFT).

A base model isn't an assistant yet

The gap is easy to miss when you use ChatGPT, Claude, Gemini or similar tools, because those models have been through several training stages. A pretrained language model starts from something much simpler: predict the next token. For instance:

Indonesia adalah negara ...

It learns what's likely to follow. That differs fundamentally from an exchange like this:

User:
Apa itu machine learning?

Assistant:
Machine learning adalah ...

The second form carries an interaction pattern: a role, an instruction and an expected reply. SFT is one technique for teaching a pretrained model that kind of behavior. The pipeline for this experiment looked like this:

Indonesian Corpus
       ↓
Pretraining
       ↓
NovAI Base 102M
       ↓
Supervised Fine-Tuning
       ↓
NovAI Chat 102M

The notable part is that the architecture stayed as it was. The parameter count remained around 102 million. What changed was what the model was trained to imitate.

Starting point

SFT began from the pretrained checkpoint, NovAI Base 102M V3. The pretraining run covered 2,000,027,648 tokens in 15,259 steps, and the logs show BF16 on CUDA during the A100 run. By its end the model was a usable language model, but what it needed now was different data entirely. More web text wouldn't help; examples of the desired responses would.

The SFT dataset

The dataset went through several iterations. The second version started from an existing set of 30,084 SFT examples and added 324 persona and factual QA examples, bringing the total to 30,408. That addition mattered. The new examples weren't general question answering. Some were written specifically to establish facts about the model itself: its identity, its role and related answers. The data was stored as conversations, conceptually like this:

User:
Apa itu machine learning?

Assistant:
Machine learning adalah ...

The model doesn't see tidy formatted conversations, though. It sees tokens, so a decision was needed about how conversations get serialized.

Teaching the conversation format

A ChatML-style structure with explicit conversation boundaries was used:

<|im_start|>
<|im_end|>

A conversation comes out approximately like this:

<|im_start|>user
Apa itu machine learning?
<|im_end|>

<|im_start|>assistant
Machine learning adalah ...
<|im_end|>

That tells the model where a message starts, where it ends and which role produced it. The tokenizer was also configured with a chat template, so the same structure can be generated in code during both training and inference. It may seem like a minor implementation detail, but it isn't. If the training format and the inference format differ, the model may have learned one conversational protocol while being asked to work with another. With a small model, keeping that interface simple and deterministic was the priority.

Dropping long conversations

Sequence length came next. For this run, the tokenizer's maximum was set to 512 tokens. The notebook tokenizes each rendered conversation without truncation and filters out examples over that limit. The filter logic was essentially:

def filter_long_sequences(example):
    text = tokenizer.apply_chat_template(
        example["messages"],
        tokenize=False
    )

    tokens = tokenizer(
        text,
        truncation=False
    )    return len(tokens["input_ids"]) <= 512

Why not just truncate? Because truncation can cut away exactly the part that matters. Slicing a training example in the middle of the assistant's answer would train the model on an incomplete response. The choice was to discard overlong sequences instead of quietly changing what they mean. It's a small engineering call, but small models make such calls count.

The training run

Training used the Hugging Face stack: PyTorch, Transformers, TRL and its SFTTrainer. The trainer was set up directly on the pretrained model and the merged SFT dataset. The effective batch size works out to:

16 × 2 = 32

Gradient accumulation was used because fitting the whole effective batch into GPU memory at once wasn't the aim. It works like this:

Batch 1
   ↓
forward / backward
Batch 2
   ↓
forward / backward
   ↓
optimizer update

The result is an effective batch of 32 without needing the GPU to hold all 32 examples at the same time. In the notebook, max_steps is derived from the dataset size and effective batch size, so the main run amounts to roughly one pass over the data and not an arbitrary large number of epochs.

The first result wasn't the end

After the first SFT stage there was a new model, NovAI Chat 102M, which could be used in a chat format. Testing showed behavior that hadn't been fully expected. The model produced answers, but not always cleanly. In one interactive test, it skipped a concise reply and drifted into unrelated text, eventually reproducing fragments such as:

<Kata-kata yang menggambarkan kehidupan seorang wanita...

and later began emitting an incomplete special token:

<|im_

That wasn't the intended behavior, and it shows why small-model experiments are useful. Strange output from a frontier model is easy to blame on someone else. With a model you built, you have to work out which part of the pipeline caused it. The dataset? The chat template? The training objective? The sampling settings? The model's limited capacity? Too few examples of the expected behavior? There wasn't necessarily a single answer, but one specific problem could be addressed.

The identity problem

The model should have a consistent identity. If someone asks:

Siapa kamu?

the reply should be predictable and about the model itself. That sounds trivial, but not for a model that mostly absorbed general language patterns. If the training data is full of various AI assistants, names, personas and conversational styles, a small model easily learns inconsistent associations.

Retraining from scratch would be overkill, so the fix was much smaller: a dedicated identity dataset and another SFT stage acting as an identity patch. The notebook labels the stage with the message PATCH TRAINING IDENTITAS (LLM-NOVA) DIMULAI. It wasn't meant to teach new general knowledge. The goal was narrow: reinforce the identity and behavior the model should associate with itself.

A second SFT stage

The second stage started from the already fine-tuned chat model and not from the original base. The pipeline became:

NovAI Base 102M
       │
       │ General SFT
       ▼
NovAI Chat 102M
       │
       │ Identity SFT
       ▼
NovAI Chat 102M + Identity

The identity stage used a more conservative configuration, on purpose. General SFT tries to change the model's conversational behavior across the board. The identity stage is much narrower, and the aim was a small behavioral adjustment, not another aggressive rewrite.

Why not fold it into the first run?

A natural question is why the identity examples weren't simply added to the main dataset and trained together. There are reasonable arguments for that. Separate stages, however, allow the effect to be isolated. Rather than this:

30K+ examples
      ↓
one opaque training run

you can reason about two stages:

General instruction data
        ↓
Chat behavior

thenIdentity examples
        ↓
Identity behavior

Splitting the stages keeps the experiment easier to read. It also reflects a lesson that keeps gaining weight with smaller models: not every problem needs another large training run. Sometimes a targeted intervention is the right answer.

The parameter count never moved

Count the parameters at each step. Before SFT: 102M. After SFT: 102M. After the identity pass: still 102M. The architecture didn't become more powerful and no billions of parameters appeared. The weights did change, and so did what they were trained on. Pretraining optimized for modeling language. SFT optimized for imitating instruction-following examples. The identity stage then pushed the model toward one specific behavioral pattern.

What the experiment shows

Don't over-read the outcome. A 102M model can't compete with GPT-class frontier models, and nothing here suggests otherwise. Nor does it prove that SFT magically improves a small model's reasoning, because it doesn't. The result is more practical: a pretrained Indonesian language model can go through another training stage that explicitly teaches it the shape of a conversation, how to answer an instruction, what a good assistant reply looks like and, via targeted examples, how to describe itself. That's a very different capability from raw next-token prediction.

SFT has limits

SFT does a lot, yet it isn't the final stage. It won't remove hallucination by itself, guarantee factual accuracy, create unlimited reasoning ability or give the model access to information outside its parameters. And it doesn't teach which answer people prefer when several are possible. Those gaps call for other techniques:

Pretraining
     ↓
SFT
     ↓
Preference Optimization
     ↓
RAG / External Knowledge
     ↓
Tools / Agents

Each layer handles a different limitation, which is why building an LLM stops looking like one training run and starts looking like a pipeline.

Why small models are interesting

What makes this work interesting isn't that a 102M model got trained. It's that the whole pipeline can be inspected, from corpus to tokenizer and onward:

Corpus
  ↓
Tokenizer
  ↓
Architecture
  ↓
Pretraining
  ↓
Checkpoint
  ↓
SFT Dataset
  ↓
Chat Template
  ↓
SFT
  ↓
Identity Patch
  ↓
Inference

When something goes wrong, you can go back and examine the layer responsible, which is very different from calling an API. Building from scratch pays off exactly there.

Build first, then improve

The question that started the series was whether Indonesia could build a foundation model of its own. A 102M model sits far below frontier scale, yet it settles a smaller version of that question: a small language model can indeed be built and trained locally. The next question was whether it could be turned into something that behaves like an assistant. SFT offered a path, and the identity experiment added a lesson: a model that can talk still has to be taught how it should behave.

What's still open

Several things remain to be tested:

• Would a stronger Indonesian instruction set noticeably lift the results?
• Does the quality of the data count for more than its volume, and by how much?
• Would preference optimization sharpen the answers?
• Could a narrow, single-domain specialist be unexpectedly useful at this size, rather than a generalist that tries to know it all?
• Where is the ceiling for a small Indonesian model when its data, objective and evaluation are all designed around Indonesian use cases?

Nobody has those answers yet, which is why the work continues. Nothing here argues that a model this size will displace frontier AI. The aim is to learn what actually happens when the whole pipeline is built by hand, stage after stage. Pretraining was milestone one and SFT milestone two.