Articles for people who
ship the stack
Rewrites on React, Node.js, TypeScript and AI. A translated body is shown when it exists. If it does not, you see the English text and a note that says so.
Tagged: transformers
Training a Small GPT on Indian Folk Tales: Tokenizer, Mask and Loss
A from-scratch 9M-parameter GPT: why GPT-2's tokenizer fails on folk-tale words, how BPE and the causal mask work, and how one batch yields 4,096 next-token losses.
16 min readHow Transformers Work, From RNN Bottlenecks to Attention, Agents and Context Windows
A step-by-step tour of the transformer: why RNNs stalled, how tokens and embeddings work, attention, QKV, heads, positions, block structure, scale and agents.
40 min readFrom Next-Token Prediction to Working Code: How LLMs Generate Python
Tokens, softmax, temperature, self-attention and feed-forward layers explained in order, with a step-by-step sorting example and the hard limits that follow from prediction.
9 min readFrom Next-Token Prediction to Instruction-Following: Fine-Tuning a GPT Model
How a pretrained GPT learns to follow instructions: formatting records, label shifting, padding masks, the training loop, evaluation limits and the step toward DPO preferences.
9 min readInside an LLM: Pure Python Inference for Qwen3-0.6B on MLX
A step-by-step build of LLM inference in about 200 lines of Python with MLX: tokenizer, RMSNorm, RoPE, grouped query attention, SwiGLU, KV cache and sampling.
13 min readLoRA and QLoRA Explained: Fine-Tuning a 7B Model Without 100 GB of VRAM
Why full fine-tuning needs 80 to 100 GB, how low-rank adapters shrink trainable weights to 1.6%, how to pick rank and target layers, and what QLoRA adds.
5 min readWhat GPT Really Stands For and Why AI Engineering Is Still Engineering
Unpack Generative Pre-trained Transformer, attention, fit and predict, and see why good AI work still depends on ordinary engineering judgment and better questions.
4 min readFrom Rules to Transformers: How NLP Learned to Handle Context
A short history of natural language processing: rule-based systems, N-grams, Markov models, word vectors, RNNs, attention, Transformers and the large language models built on them.
7 min readWhy Transformers Use Sine and Cosine for Positional Encoding
A step-by-step explanation of sinusoidal positional encoding, from why attention ignores word order to clocks at different speeds, a worked 12-dimension example and rotation.
8 min read
About these articles
Request a 24h estimate
Need the same stack in a production operator layer? Send the brief — estimate within 24 hours.