Articles for people who
ship the stack
Rewrites on React, Node.js, TypeScript and AI. A translated body is shown when it exists. If it does not, you see the English text and a note that says so.
Tagged: Tokenization
Copy Alone, Fail in Company: Contextual Fragility Among Tokens That Pass an Isolated Copying Test
Screening eight open-weight models for tokens that copy correctly alone but are dropped or altered in context, with checks on embedding geometry, task preservation and tool use.
32 min readFrom Next-Token Prediction to Working Code: How LLMs Generate Python
Tokens, softmax, temperature, self-attention and feed-forward layers explained in order, with a step-by-step sorting example and the hard limits that follow from prediction.
9 min readFragile Tokens and Silent Substitution: A Geometric Account of Some Hallucinations
Some hallucinations come from damaged or fragile tokens that get swapped for near neighbors before reasoning starts. Examples, per-model rates and a simple test.
7 min readFrom Prompt to Token: How an LLM Turns Text Into Numbers and Back
Follow a short prompt through tokenization, embeddings, attention, feed-forward layers, softmax and sampling to see how a model really picks each next word.
7 min readTokenization Is a System Design Decision, Not a Preprocessing Step
How byte-level BPE builds the vocabulary your LLM sees, why token counts differ by language, which failures it causes, and what to measure before blaming the model.
9 min readWhy Indian Languages Cost More Tokens in LLMs and How Tokenizers Get Fixed
The same sentence takes 5 tokens in English and 34 in Tamil. See how BPE training data causes it, what rebalancing and vocabulary size change, and how vocabulary expansion works.
8 min readHow Embedding Layers Work: From Text to Token IDs to Vectors in an LLM
A beginner-friendly look at embedding layers: tokenization, lookup tables, learned vectors, the 205 million parameter example, and a short PyTorch demo with nn.Embedding.
5 min readTraining a Small LLM From Scratch on an 8 GB RAM Laptop
How to set up a clean data pipeline for training a transformer on modest hardware: BPE tokenizer, TinyStories, disk-backed tokens and next-token batching.
5 min readToken Budgets vs. Semantic Boundaries: Getting RAG Chunk Splits Right
Learn how tokenization and chunking differ, where each sits in a RAG pipeline, and how to split documents by meaning while measuring size with the real tokenizer.
11 min read
About these articles
Request a 24h estimate
Need the same stack in a production operator layer? Send the brief — estimate within 24 hours.