Master transformer architecture by building your own GPT

Multi-dimensional arrays as the foundation of every model computation — shapes, strides, and indexing.

Element-wise math, scalar operations, and the building blocks that power every layer.

Row-major order, cache lines, and why contiguous access is ~100x faster than random access.

The single most important operation in deep learning — naive implementation and why it matters.

Loop reordering, cache blocking, and going from 0.5 to 5 GFLOPS on a single CPU core.

Extending matmul to handle batches — essential for multi-head attention.

The fundamental building block: output = input × weight + bias, plus the activation that replaced ReLU.

Stabilizing activations with mean/variance normalization and learned scale-shift parameters.

Turning raw logits into probability distributions without overflow or underflow.

Queries, keys, values — the mechanism that lets tokens communicate with each other.

Parallel attention heads and the mask that prevents looking into the future.

The two-layer MLP where the model processes the information attention gathered.

Assembling attention + FFN with skip connections and pre-norm into a complete block.

Converting text to numbers and back — character-level encoding and the path to BPE.

Reading pretrained parameters from binary files and placing them into the right structures.

Token and position embeddings, stacking transformer blocks, and producing logits end-to-end.

Temperature, top-k, and nucleus sampling — controlling the randomness of generation.

The generation loop: predict one token, append it, repeat — building a working inference engine.

The key optimization for inference — caching keys and values to avoid redundant computation.

Finding bottlenecks, parallelizing with threads, and pre-allocating memory for speed.

RoPE — encoding position directly into attention via rotation, as used in LLaMA and Mistral.