Schedule

Module 1 - Introduction

Aug 25

Introduction to the course: Post lecture activity: Make an account on the Perlmutter supercomputer

Aug 27

Introduction to transformers: Optional reading: (1) Attention is all you need (2) The Illustrated Transformer

Module 2 - Systems implications of the Transformer architecture

Sep 1

No Class (Labor Day)

Sep 3

Memory, Compute and Communication needs of LLMs: Optional reading:

Module 3 - ML Infrastructure (Part 1)

Sep 8

Multi-GPU servers and interconnects

Sep 10

Training an LLM Part 1 (hands-on activity): [Bring your laptops to class]

Module 4 - ML Infrastructure (Part 2)

Sep 15

ML-centric Datacenters

Sep 17

Training an LLM Part 2 (hands-on activity): [Bring your laptops to class]

Module 5 - Distributed Training and Finetuning

Sep 22: Introduction to distributed training
Sep 24: Hybrid Parallelism (Part 1)
Sep 29: Hybrid Parallelism (Part 2)
Oct 01: Collective Communication