Schedule

Module 1 - Introduction

Module 2 - Systems implications of the Transformer architecture

Module 3 - Hardware infrastructure for machine learning

Mon, Sep 21
AI infrastructure I: from one GPU to many
Using the roofline model, communication as the third resource, the bandwidth hierarchy, scale-up versus scale-out, xAI’s Colossus
Optional reading: xAI’s Colossus 2: First Gigawatt Datacenter in the World (SemiAnalysis)
Wed, Sep 23
AI infrastructure II: scale-up domains and TPUs
NVIDIA DGX servers, NVLink and NVSwitch, GB200 NVL72, the TPU chip and its systolic array
Optional reading: (1) NVIDIA Hopper Architecture In-Depth (2) NVIDIA GB200 NVL72 (3) How to Scale Your Model, How to Think About TPUs
Mon, Sep 28
AI infrastructure III: TPU pods and scale-out networks
TPU generations, the ICI torus and optical circuit switches, Clos and rail-optimized fabrics
Optional reading: (1) TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning (2) A Scalable, Commodity Data Center Network Architecture (3) NVIDIA DGX SuperPOD reference architecture: network fabrics
Wed, Sep 30
AI infrastructure IV: RDMA
Why not TCP, registered memory and queue pairs, InfiniBand and RoCE, priority flow control (PFC), what breaks at 100,000 GPUs, OpenAI’s Multipath Reliable Connection (MRC)
Optional reading: (1) RDMA lecture slides (Radhika Mittal, UIUC) (2) RDMA over Ethernet for Distributed AI Training at Meta Scale (3) The Multipath Reliable Connection (MRC) Transport
Try it on your own
Train a small LLM from scratch
Not a class meeting; work through the notebook before Module 4 starts

Module 4 - Distributed training with data parallelism

Mon, Oct 5
Introduction to distributed training and data parallelism
Finishing RDMA (how OpenAI’s MRC spreads traffic over many paths and recovers from loss, MRC in production), why the network matters for training, model FLOPs utilization (MFU), four ways to split a training step, data parallelism with the AllReduce, overlapping the AllReduce with the backward pass
Optional reading: (1) PyTorch Distributed: Experiences on Accelerating Data Parallel Training (2) Resilient AI Supercomputer Networking using MRC and SRv6 (3) The Ultra-Scale Playbook: Training LLMs on GPU Clusters (Hugging Face), data parallelism section
Wed, Oct 7
Finishing data parallelism, and ZeRO-1
No in-person class: watch the recorded lecture. When the backward pass hides the AllReduce (tokens per GPU), bucketing gradients, gradient accumulation, three limits of data parallelism, 16 bytes of training state per parameter, ReduceScatter and AllGather, ZeRO-1 (sharding the optimizer state)
Optional reading: (1) Mixed Precision Training (2) An Empirical Model of Large-Batch Training (3) ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Mon, Oct 12
No Class (Fall Break)
Wed, Oct 14
ZeRO-2, ZeRO-3 and FSDP
Each GPU owns a fixed part of the parameters, ZeRO-2 (sharding the gradients), ZeRO-3 (sharding the weights), memory per GPU for a 7.5B-parameter model on 64 GPUs, PyTorch Fully Sharded Data Parallel (FSDP) and prefetching the AllGathers
Optional reading: PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Module 5 - Distributed training with tensor, pipeline, sequence and expert parallelism

Mon, Oct 19
Finishing ZeRO, and tensor parallelism
FSDP in PyTorch (fully_shard, hybrid sharding), the cost of each ZeRO stage, the limits of ZeRO, splitting a matrix multiply by columns and by rows, tensor parallelism in the MLP and in attention (Megatron-LM), four AllReduces per layer, why throughput per GPU drops past one server
Optional reading: (1) Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2) Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (3) PyTorch documentation, fully_shard (FSDP2)
Wed, Oct 21
Sequence and pipeline parallelism
Memory with tensor parallelism, tensor parallelism versus ZeRO-3, sequence parallelism (splitting LayerNorm and dropout along the tokens), pipeline parallelism: the bubble, micro-batches, and the all-forward-all-backward (AFAB) and one-forward-one-backward (1F1B) schedules
Optional reading: (1) Reducing Activation Recomputation in Large Transformer Models (2) GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (3) PipeDream: Generalized Pipeline Parallelism for DNN Training
Mon, Oct 26
Finishing pipeline parallelism, and mixture of experts
Shrinking the pipeline bubble (interleaved stages, DualPipe), pipeline parallelism versus ZeRO-3, mixture-of-experts layers (more parameters without more compute per token), routing tokens to the top-k experts, why routing is often unbalanced
Optional reading: (1) Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2) Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (3) DeepSeek-V3 Technical Report (DualPipe)
Wed, Oct 28
Expert parallelism, context parallelism and collectives in practice
Balancing the load across experts, expert parallelism with all-to-all dispatch and combine, case studies (Mixtral 8x7B, DeepSeek-V3), context parallelism with ring attention, putting the five parallelisms together, collectives in practice (ring AllReduce, switches that add, vLLM’s custom AllReduce, DeepEP)
Optional reading: (1) GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2) Ring Attention with Blockwise Transformers for Near-Infinite Context (3) DeepEP: an efficient expert-parallel communication library (DeepSeek)

Module 6 - Guest lectures

Mon, Nov 2
Guest lecture: Abhinav Jangda (Microsoft Research)
SubCubic-LLM: Faster LLM Inference using Strassen-like Algorithms for Matrix Multiplication (abstract and bio)
Wed, Nov 4
Guest lecture: Benjamin Klenk (NVIDIA Research)
From the Chip to the Rack and Beyond: Architecture for Large Language Models (abstract and bio)

Project mid-point presentations

Mon, Nov 9
Project mid-point presentations

Module 7 - Inference and post-training

Wed, Nov 11
LLM inference optimizations
KV caching, batching, speculative decoding
Mon, Nov 16
Post-training LLMs
Supervised finetuning, parameter-efficient finetuning, reinforcement learning from feedback

Module 8 - Agents, harnesses and world models

Wed, Nov 18
LLM agents
Mon, Nov 23
Agent harnesses
Tool use, sandboxes, evaluation of agentic systems
Wed, Nov 25
No Class (Thanksgiving Break)
Mon, Nov 30
Guest lecture: To Be Announced

Module 9 - Wrap-up and final poster session

Wed, Dec 2
Course wrap-up
End-of-semester survey
Mon, Dec 7
Final project poster session
Last day of instruction