ML Notes · 10 posts

Writing

I've been fascinated by the emergent capabilities that appear as we scale generative models. It surprised me that a simple objective like next-word prediction can produce something that behaves like a world model, and I wanted to understand exactly how that happens at the implementation level. So I worked through Andrej Karpathy's Zero to Hero series, coding each lecture rather than just watching, and documented my progress in this technical blog.

The blog builds the language model stack from first principles: an autograd engine, a bigram character-level model, an MLP, batch normalization, a WaveNet-inspired model, a character-level transformer, a BPE tokenizer, and finally GPT-2 pretrained on 4 A100 GPUs.

Mar 20, 2026 Reproducing GPT-2 on 4 A100 GPUs Full reproduction of the 124M parameter GPT-2 from scratch — bfloat16, torch.compile, FlashAttention, gradient accumulation, DDP across 4 A100s. Trained on FineWeb-Edu 10B; beats the reference eval score on HellaSwag. read → Feb 21, 2026 Tokenization Understanding how tokenization works in LLMs — Byte Pair Encoding, GPT-2/TikToken patterns, special tokens, and why tokenization explains many LLM quirks. read → Feb 19, 2026 Building GPT Building a character-level language model from scratch using transformers — from the simplest bigram model through self-attention, multi-head attention, feed forward networks, residual connections, and layer norm. read → Aug 28, 2025 Building a WaveNet-Inspired MLP Architecture A deep dive into building a WaveNet-inspired MLP architecture, exploring hierarchical structures and improvements to the model. read → Aug 18, 2025 Backprop on Tensors: Understanding PyTorch's Autograd Engine A deep dive into the inner workings of the backward pass in neural networks, implementing manual backpropagation at the tensor level to match PyTorch's autograd. read → Jul 22, 2025 Batch Normalization Deep Dive A deep dive into Batch Normalization, a technique to stabilize and accelerate the training of deep neural networks. read → Jul 14, 2025 MLP Language Model A character-level language model using a multi-layer perceptron to predict the next character in a sequence. read → Jul 4, 2025 Bigram Character-Level Language Model An exploration of building a bigram character-level language model using both frequency counting and a neural network approach. read → Jul 3, 2025 Autograd Engine Deep Dive A deep dive into building a simple autograd engine for neural networks, explaining automatic differentiation and backpropagation. read → Jun 8, 2025 Hugging Face Transformer's Library A quick introduction to the Hugging Face Transformers library and how to get started with state-of-the-art NLP models. read →