Schedule

Module 1 - Introduction

Mon, Aug 24

Introduction to the course

Post-lecture activity: Make an account on the Perlmutter supercomputer

Wed, Aug 26

Introduction to transformers

Optional reading: (1) Attention is all you need (2) The Illustrated Transformer

Module 2 - Systems implications of the Transformer architecture

Mon, Aug 31

Memory use in Transformer-based LLMs

Optional reading: Reducing Activation Recomputation in Large Transformer Models

Wed, Sep 2

Performance of ML workloads: compute, data movement and roofline thinking

Optional reading: (1) Roofline: An Insightful Visual Performance Model (2) Making Deep Learning Go Brrrr From First Principles

Mon, Sep 7

No Class (Labor Day)

Wed, Sep 9

Class cancelled

Mon, Sep 14

Counting FLOPs in a Transformer, arithmetic intensity and the roofline model

Optional reading: How to Scale Your Model, chapters 1 (rooflines) and 4 (Transformer math)

Module 3 - Hardware infrastructure for machine learning

Wed, Sep 16

AI infrastructure I: scale-up domains

NVLink and NVSwitch, GB200 NVL72, TPU pods and optical circuit switches, Cerebras wafer-scale systems

Mon, Sep 21

AI infrastructure II: datacenter networks and RDMA

Clos and rail-optimized fabrics, RoCE and InfiniBand, PFC, IRN, OpenAI’s Multipath Reliable Connection (MRC)

Try it on your own

Train a small LLM from scratch

Not a class meeting; work through the notebook before Module 4 starts

Module 4 - Distributed training with data parallelism

Wed, Sep 23

Introduction to distributed training

Mon, Sep 28

Data parallelism with ZeRO

Wed, Sep 30

Data parallelism with ZeRO-3 and FSDP

Mon, Oct 5

Collective communication

All-reduce algorithms, NCCL