Skip to main content Link Expand (external link) Document Search Copy Copied Schedule Module 1 - Introduction Module 3 - Hardware infrastructure for machine learning Mon, Sep 21 AI infrastructure I: from one GPU to many Using the roofline model, communication as the third resource, the bandwidth hierarchy, scale-up versus scale-out, xAI’s Colossus Optional reading: xAI’s Colossus 2: First Gigawatt Datacenter in the World (SemiAnalysis) Wed, Sep 23 AI infrastructure II: scale-up domains and TPUs NVIDIA DGX servers, NVLink and NVSwitch, GB200 NVL72, the TPU chip and its systolic array Optional reading: (1) NVIDIA Hopper Architecture In-Depth (2) NVIDIA GB200 NVL72 (3) How to Scale Your Model, How to Think About TPUs Mon, Sep 28 AI infrastructure III: TPU pods and scale-out networks TPU generations, the ICI torus and optical circuit switches, Clos and rail-optimized fabrics Optional reading: (1) TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning (2) A Scalable, Commodity Data Center Network Architecture (3) NVIDIA DGX SuperPOD reference architecture: network fabrics Wed, Sep 30 AI infrastructure IV: RDMA Why not TCP, registered memory and queue pairs, InfiniBand and RoCE, priority flow control (PFC), what breaks at 100,000 GPUs, OpenAI’s Multipath Reliable Connection (MRC) Optional reading: (1) RDMA lecture slides (Radhika Mittal, UIUC) (2) RDMA over Ethernet for Distributed AI Training at Meta Scale (3) The Multipath Reliable Connection (MRC) Transport Try it on your own Train a small LLM from scratch Not a class meeting; work through the notebook before Module 4 starts Module 4 - Distributed training with data parallelism Mon, Oct 5 Introduction to distributed training and data parallelism Finishing RDMA (how OpenAI’s MRC spreads traffic over many paths and recovers from loss, MRC in production), why the network matters for training, model FLOPs utilization (MFU), four ways to split a training step, data parallelism with the AllReduce, overlapping the AllReduce with the backward pass Optional reading: (1) PyTorch Distributed: Experiences on Accelerating Data Parallel Training (2) Resilient AI Supercomputer Networking using MRC and SRv6 (3) The Ultra-Scale Playbook: Training LLMs on GPU Clusters (Hugging Face), data parallelism section Wed, Oct 7 Finishing data parallelism, and ZeRO-1 No in-person class: watch the recorded lecture . When the backward pass hides the AllReduce (tokens per GPU), bucketing gradients, gradient accumulation, three limits of data parallelism, 16 bytes of training state per parameter, ReduceScatter and AllGather, ZeRO-1 (sharding the optimizer state) Optional reading: (1) Mixed Precision Training (2) An Empirical Model of Large-Batch Training (3) ZeRO: Memory Optimizations Toward Training Trillion Parameter Models Mon, Oct 12 No Class (Fall Break) Wed, Oct 14 ZeRO-2, ZeRO-3 and FSDP Each GPU owns a fixed part of the parameters, ZeRO-2 (sharding the gradients), ZeRO-3 (sharding the weights), memory per GPU for a 7.5B-parameter model on 64 GPUs, PyTorch Fully Sharded Data Parallel (FSDP) and prefetching the AllGathers Optional reading: PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel Module 5 - Distributed training with tensor, pipeline, sequence and expert parallelism Mon, Oct 19 Finishing ZeRO, and tensor parallelism FSDP in PyTorch (fully_shard, hybrid sharding), the cost of each ZeRO stage, the limits of ZeRO, splitting a matrix multiply by columns and by rows, tensor parallelism in the MLP and in attention (Megatron-LM), four AllReduces per layer, why throughput per GPU drops past one server Optional reading: (1) Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2) Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (3) PyTorch documentation, fully_shard (FSDP2) Wed, Oct 21 Sequence and pipeline parallelism Memory with tensor parallelism, tensor parallelism versus ZeRO-3, sequence parallelism (splitting LayerNorm and dropout along the tokens), pipeline parallelism: the bubble, micro-batches, and the all-forward-all-backward (AFAB) and one-forward-one-backward (1F1B) schedules Optional reading: (1) Reducing Activation Recomputation in Large Transformer Models (2) GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (3) PipeDream: Generalized Pipeline Parallelism for DNN Training Mon, Oct 26 Finishing pipeline parallelism, and mixture of experts Shrinking the pipeline bubble (interleaved stages, DualPipe), pipeline parallelism versus ZeRO-3, mixture-of-experts layers (more parameters without more compute per token), routing tokens to the top-k experts, why routing is often unbalanced Optional reading: (1) Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2) Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (3) DeepSeek-V3 Technical Report (DualPipe) Wed, Oct 28 Expert parallelism, context parallelism and collectives in practice Balancing the load across experts, expert parallelism with all-to-all dispatch and combine, case studies (Mixtral 8x7B, DeepSeek-V3), context parallelism with ring attention, putting the five parallelisms together, collectives in practice (ring AllReduce, switches that add, vLLM’s custom AllReduce, DeepEP) Optional reading: (1) GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2) Ring Attention with Blockwise Transformers for Near-Infinite Context (3) DeepEP: an efficient expert-parallel communication library (DeepSeek) Module 6 - Guest lectures Mon, Nov 2 Guest lecture: Abhinav Jangda (Microsoft Research) SubCubic-LLM: Faster LLM Inference using Strassen-like Algorithms for Matrix Multiplication (abstract and bio ) Wed, Nov 4 Guest lecture: Benjamin Klenk (NVIDIA Research) From the Chip to the Rack and Beyond: Architecture for Large Language Models (abstract and bio ) Project mid-point presentations Mon, Nov 9 Project mid-point presentations Module 7 - Inference and post-training Wed, Nov 11 LLM inference optimizations KV caching, batching, speculative decoding Mon, Nov 16 Post-training LLMs Supervised finetuning, parameter-efficient finetuning, reinforcement learning from feedback Module 8 - Agents, harnesses and world models Wed, Nov 18 LLM agents Mon, Nov 23 Agent harnesses Tool use, sandboxes, evaluation of agentic systems Wed, Nov 25 No Class (Thanksgiving Break) Mon, Nov 30 Guest lecture: To Be Announced Module 9 - Wrap-up and final poster session Wed, Dec 2 Course wrap-up End-of-semester survey Mon, Dec 7 Final project poster session Last day of instruction