02Inside the Chip

Inside the Chip

BasicModule 025 topics~4h 10m

Topics, in order

01 SMs, Warps and SIMTA GPU chip is a grid of identical blocks called streaming multiprocessors (SMs). Each SM runs many threads, but not one at a time: it groups them into bundles of 32 called warps and gives … Beginner 1h 02 The Memory HierarchyData on a GPU can live in several places. The closer to the arithmetic units, the faster and the smaller. From nearest to furthest: registers, shared memory / L1, L2 cache, global memory … Beginner 50 min 03 HBM and Memory BandwidthData-center GPUs use a special memory called HBM (high-bandwidth memory): memory chips stacked vertically and placed right next to the GPU, connected by thousands of microscopic wires. The … Intermediate 45 min 04 Tensor Cores and Number FormatsA tensor core is a circuit inside each SM that does one thing: multiply two small matrices and add the result to a third, in a single operation. Because it is built for that one job and … Intermediate 55 min 05 Power, Heat and the Host LinkA GPU does not float in space. It is attached to a host through PCIe, it draws hundreds of watts, and every one of those watts becomes heat that must be removed. These "boring" physical … Intermediate 40 min

About this module

Module I said a GPU has “thousands of lanes” and “fast memory”. This module opens the lid and names every part, using NVIDIA’s H100 as the worked example.

#	Lesson	The question it answers
01	SMs, Warps and SIMT	How are thousands of lanes organized and controlled?
02	The Memory Hierarchy	Where can data live on a GPU, and how fast is each place?
03	HBM and Memory Bandwidth	What is the memory made of, and why is its speed the headline?
04	Tensor Cores and Number Formats	Where do the giant FLOP numbers come from?
05	Power, Heat and the Host Link	What connects the chip to the rest of the world?

When you finish you can draw a GPU from memory — SMs, warps, registers, shared memory, caches, HBM, PCIe — and attach a rough speed and size to every box.