Articles for people who
ship the stack
Rewrites on React, Node.js, TypeScript and AI. A translated body is shown when it exists. If it does not, you see the English text and a note that says so.
Tagged: PyTorch
Training a Small GPT on Indian Folk Tales: Tokenizer, Mask and Loss
A from-scratch 9M-parameter GPT: why GPT-2's tokenizer fails on folk-tale words, how BPE and the causal mask work, and how one batch yields 4,096 next-token losses.
16 min readFrom Next-Token Prediction to Instruction-Following: Fine-Tuning a GPT Model
How a pretrained GPT learns to follow instructions: formatting records, label shifting, padding masks, the training loop, evaluation limits and the step toward DPO preferences.
9 min readResidual Connections Explained: Why Deep Transformers Can Train at All
Why stacking layers breaks training, how x + F(x) skip connections fix gradient flow, how Pre-LayerNorm blocks use them, and a short PyTorch implementation.
4 min readBuild a Transformer From Scratch in PyTorch: Attention, Encoder and Decoder
Step-by-step PyTorch code for the core Transformer pieces: positional encoding, scaled dot-product and multi-head attention, feed-forward layers, encoder stack and decoder basics.
8 min readHow Embedding Layers Work: From Text to Token IDs to Vectors in an LLM
A beginner-friendly look at embedding layers: tokenization, lookup tables, learned vectors, the 205 million parameter example, and a short PyTorch demo with nn.Embedding.
5 min readBPE Tokenizer From Scratch: Shorter Sequences for a Small GPT Context Window
Write a Byte Pair Encoding tokenizer in Python, train it on WikiText-2, save merges as JSON, and see how subword tokenization stretches a GPT context window.
26 min readDebugging a Small GPT in PyTorch: Tests That Isolate Each Failure
A stage-by-stage workflow for debugging a character-level GPT in PyTorch, from token IDs and target shifting to gradients, NaN losses and checkpoints.
30 min read
About these articles
Request a 24h estimate
Need the same stack in a production operator layer? Send the brief — estimate within 24 hours.