Training a Small LLM From Scratch on an 8 GB RAM Laptop

Plenty of people assume you need a rack of GPUs before you can train a language model from scratch. You don't, at least not for learning. A budget gaming laptop with an Intel i5-9300H, 8 GB of RAM and a GTX 1650 holding 4 GB of VRAM is enough to build a small transformer, feed it a toy dataset and watch it improve. It will be slow and a bit cramped. It will also teach you more than any hosted API.

Here is the first stage of such a build: the data pipeline. The model itself comes later, so for now the work is tokenizers, datasets, memory limits and batching.

Why bother

There are three decent reasons to take this on:

• Python gets rusty when an assistant writes everything for you, and rebuilding the habit by hand is a good fix.
• The systems behind the current hype feel less mysterious once you've assembled one piece by piece.
• Implementing papers yourself is the shortest road to reading research critically, and maybe eventually producing some.

The project here is called Axiom, after the word for a rule accepted as true without proof. It suits a first step: write a transformer yourself, then train it, starting from the plain basics.

Following a lecture without copying it

Andrej Karpathy's video series on building GPT is the natural guide. His lecture begins with a character-level tokenizer. If you already understand tokenization, you can skip ahead and use byte-pair encoding (BPE) from the start. The point is to learn, so it makes sense to pick a slightly harder route than the one on screen, and to avoid retyping his exact implementation.

A project, not a notebook

Notebooks are comfortable for experiments, but they blur structure. Someone who lives in Neovim has an extra reason to drop them. Build it as an ordinary Python project instead, with error handling, a data loader and every step as a named function. Each notebook cell becomes one function, and a small orchestrator calls them in order.

A few choices keep this tidy:

1. A .local directory holds everything heavy that shouldn't be committed: raw datasets, tokenizers and token files already converted. Its purpose is to copy the best part of a notebook session, where nothing has to be reloaded on every run.
2. One config file records model settings, paths and tokenizer names, so nothing is hardcoded in the pipeline.
3. Each download step checks whether the file already exists before fetching it again.

Data and tokenizer

For text, the dataset is roneneldan/TinyStories from Hugging Face. It's small without being trivially tiny, and its simple vocabulary suits a model that has to learn from limited compute. An assistant suggested it, so the choice wasn't researched deeply, but it's a reasonable pick for a first run. A downloader function pulls it down once and caches it.

The tokenizer works the same way. Here it's lakhera2023/gemma4-nano-tinystories, loaded through another small conditional downloader. Again, a suggestion taken on trust and then coded by hand, which is a fair way to learn: let an assistant propose, then write the code yourself so you understand what it does.

The memory wall

Tokenization is where an 8 GB machine pushes back. The obvious approach is to tokenize every story, append the ids to one growing array in memory, and write the array out when it's done. On a laptop with limited RAM, that approach freezes the whole system.

The reason is simple once you look at it. A token is stored as an integer, and a list of integers occupies far more space than the plain text it came from. Python's list overhead makes it worse, and repeatedly extending a big structure keeps fragmenting memory. Millions of stories later, nothing is left for the operating system.

The fix is to stop holding the output at all. Write each story's tokens straight to a file on disk as soon as they're produced:

with open(output_file, "wb") as f:
        for sample in dataset:
            text = sample["text"]
            tokens = tokenizer.encode(text)
            tokens.append(tokenizer.eos_id())
            np.asarray(tokens, dtype=np.uint16).tofile(f)

Memory use is now flat, no matter how large the dataset grows. Tokens land in a binary file that can be mapped back in later, which is also how larger training pipelines work. It's the only real choke point in this stage, and finding it by crashing is a memorable lesson.

Building batches

Batching is simpler than it sounds, though it needs some mental warm-up. A language model learns to predict the next token, so the target for any input is that same sequence moved forward by one position. Take a slice of tokens as x, take the slice starting one position later as y, and stack many such pairs into a batch. For every position in x, the model sees what comes next in y.

That shifted-by-one relationship is the whole trick behind the training signal. It looks almost too easy, which is why it's worth implementing manually once instead of using a library call.

What comes next

The data side is done: a cached dataset, a cached tokenizer, a token file on disk and a batcher. The next stage is the model itself, a transformer written from scratch, still following Karpathy's lecture and deviating where there's a reason to. A laptop this small will limit model size and context length, so the real work will be fitting something useful inside 4 GB of VRAM.