SIMD and Vectorization

1. What is it?#

SIMD = Single Instruction, Multiple Data. One instruction operates on a whole vector of values at once.

Scalar:                       SIMD (AVX-512, 16 floats):
  add a[0], b[0] → c[0]         vaddps zmm0, zmm1, zmm2
  add a[1], b[1] → c[1]           → 16 additions, one instruction, one cycle
  ... 16 instructions

Vectorization is the act of turning scalar code into SIMD code — by the compiler (auto-vectorization), by a library, or by hand with intrinsics.

This is the CPU’s version of the idea that GPUs take to an extreme. Understanding SIMD makes the GPU’s SIMT execution model (Section VI.02) immediately familiar.

2. Why does it exist?#

Fetching and decoding instructions costs power and die area. If you’re going to do the same operation to a million values, doing it 16 at a time amortizes that cost 16x. It is the cheapest form of parallelism available: no threads, no synchronization, no cache coherence traffic.

Vector width has grown steadily:

SSE      (1999)   128-bit    4 floats
AVX      (2011)   256-bit    8 floats
AVX-512  (2016)   512-bit   16 floats
AMX      (2023)   tiles      matrix multiply in one instruction (Intel)
SVE/SVE2 (ARM)    scalable   128-2048 bit
NEON     (ARM)    128-bit    4 floats

A CPU core without SIMD gets ~1/16th of its peak FLOP/s. Almost all of a CPU’s advertised FLOP/s comes from SIMD.

3. Simple analogy#

A pastry chef with a 16-cavity mould. Making cupcakes one at a time is one motion per cupcake. With the mould, one pour fills 16. Same motion, 16x output.

The constraints match SIMD’s constraints exactly:

• All 16 must be the same recipe (single instruction).
• The batter must be laid out to pour in one go (contiguous, aligned data).
• If you only have 5 cupcakes to make, 11 cavities are wasted (tail handling).
• If each cupcake needs a different topping, the mould doesn’t help (divergence).

4. Tiny example#

// Scalar
void add_scalar(float* a, float* b, float* c, int n) {
    for (int i = 0; i < n; i++) c[i] = a[i] + b[i];
}

// AVX2 (8 floats at a time), explicit
#include <immintrin.h>
void add_avx2(float* a, float* b, float* c, int n) {
    int i = 0;
    for (; i + 8 <= n; i += 8) {
        __m256 va = _mm256_loadu_ps(a + i);
        __m256 vb = _mm256_loadu_ps(b + i);
        _mm256_storeu_ps(c + i, _mm256_add_ps(va, vb));
    }
    for (; i < n; i++) c[i] = a[i] + b[i];   // tail
}

Compile the scalar version with -O3 -march=native and the compiler will usually produce the AVX2 version for you. Check:

gcc -O3 -march=native -fopt-info-vec add.c -c
# reports "loop vectorized" or explains why not

But this particular loop is memory-bound (1 FLOP per 12 bytes), so SIMD gains almost nothing — you’re bandwidth-limited either way. SIMD helps when you’re compute-bound:

// Compute-bound: many FLOPs per byte
for (int i = 0; i < n; i++) {
    float x = a[i];
    for (int k = 0; k < 50; k++) x = x * 1.0001f + 0.5f;
    c[i] = x;
}

Here AVX-512 gives close to a 16x speedup. Same lesson as Section I.07: know your regime before optimizing.

5. Technical explanation#

What blocks auto-vectorization#

1. Loop-carried dependencies:    for(i) a[i] = a[i-1] + 1;     ← inherently serial
2. Possible aliasing:            void f(float* a, float* b)     ← may overlap; use restrict
3. Data-dependent branches:      if (a[i] > 0) ...              ← needs masking; often fine now
4. Non-contiguous access:        a[i*stride]                    ← gather; slow
5. Function calls in the loop    ← unless inlined
6. Unknown trip count with complex exit conditions

Fixes: restrict pointers, #pragma omp simd, align data to 64 bytes, make the trip count a multiple of the vector width, and avoid calling out of the loop.

FMA — the instruction that defines peak FLOPs#

Fused multiply-add computes a*b + c in one instruction with one rounding:

AVX-512 FMA:  16 floats × (1 multiply + 1 add) = 32 FLOPs per instruction
2 FMA ports × 32 FLOPs × 3.0 GHz = 192 GFLOP/s per core
× 32 cores = 6.1 TFLOP/s peak FP32

That is where CPU peak FLOP numbers come from. Achieving them requires FMA-dense, cache-resident code — i.e. a well-written GEMM. Ordinary code gets a small fraction.

AMX — matrix instructions on CPU#

Intel’s Advanced Matrix Extensions add 2D “tile” registers and a TDPBF16PS-style instruction that does a whole small matrix multiply. This is the CPU catching up to tensor cores. With AMX, BF16/INT8 CPU inference for small models became genuinely competitive — worth knowing when evaluating CPU serving for the long tail of small models (Section XII.10).

The GPU connection#

CPU SIMD:  one instruction, 16 lanes, one thread, explicit in the ISA
GPU SIMT:  one instruction, 32 lanes (a warp), 32 "threads", implicit

They are the same hardware idea with a different programming model. A warp is a 32-wide SIMD unit where each lane is presented to the programmer as a thread. Warp divergence (Section VI.10) is exactly the “different toppings” problem from the analogy, handled by masking.

6. Under the hood#

Check what your binary actually uses:

# Does this binary contain AVX-512 instructions?
objdump -d ./mybinary | grep -c "zmm"

# What does the CPU support?
lscpu | grep -o 'avx[0-9_a-z]*' | sort -u
grep -o 'amx[_a-z]*' /proc/cpuinfo | sort -u

# What is NumPy/PyTorch using?
python -c "import numpy; numpy.show_config()"
python -c "import torch; print(torch.__config__.show())"

For inference on CPU, the vectorized paths live in oneDNN (PyTorch CPU backend), OpenBLAS/MKL, or hand-written kernels (llama.cpp has per-ISA implementations). You almost never write SIMD by hand in this field — you make sure the right library path is selected.

7. Performance implications#

Where SIMD shows up in inference:

8. Production implications#

• Build for the target ISA. A container built with -march=x86-64 baseline runs 3-10x slower on CPU numeric paths than one built with -march=native or explicit AVX-512 dispatch. Most ML libraries do runtime dispatch, but not all of your code does.
• Check that MKL/oneDNN picked the right kernels. MKLDNN_VERBOSE=1 or DNNL_VERBOSE=1 prints which implementation ran (e.g. avx512_core_amx vs ref). Seeing ref in a hot path means you’re running the reference C implementation — 10-50x slow.
• AVX-512 frequency offsets can reduce all-core clocks. On older Skylake-SP, heavy AVX-512 could slow down other threads on the socket. Mostly resolved on newer parts, but measure end-to-end, not just the kernel.
• For gateway/CPU-heavy nodes, use simdjson-class parsers and Rust/C tokenizers.

9. Common mistakes#

Hand-writing intrinsics before checking the compiler already vectorized. Look at -fopt-info-vec first.

Vectorizing a memory-bound loop. No gain. See section 4.

Forgetting alignment and tails. Unaligned loads are cheap on modern x86 but not free; tails can dominate for short loops.

Shipping a generic build. Especially in containers, where “portable” is the default.

Assuming AVX-512 is always faster than AVX2. On some parts with frequency offsets and for memory-bound code, AVX2 wins. Measure.

10. Hands-on exercise#

A. Check your ISA. Determine which vector ISAs your CPU supports and which your PyTorch build uses.

B. Measure the ceiling. Write an FMA-dense microbenchmark (a long chain of independent x = x*a + b on 8 or 16 accumulators to fill the ports) and measure GFLOP/s per core. Compare to cores × 2 FMA × width × 2 × GHz. What fraction do you achieve?

C. Auto-vectorization report. Take a loop that doesn’t vectorize (add a loop-carried dependency), compile with -fopt-info-vec-missed, read the reason, then fix it.

D. oneDNN verbose. Run a small transformer on CPU with DNNL_VERBOSE=1. Identify which kernels ran and whether any fell back to a reference implementation.

11. Interview questions#

1. What is SIMD and how does it relate to a GPU warp?
2. Why doesn’t SIMD help a memory-bound loop?
3. Name four things that prevent auto-vectorization.
4. Where does a CPU’s peak FLOP/s number come from? Derive it.
5. What is AMX and why does it matter for CPU inference?
6. How would you verify that your production container is using AVX-512 paths?

12. Further reading#

• [REFERENCE] Intel Intrinsics Guide — https://www.intel.com/content/www/us/en/docs/intrinsics-guide/
• [REFERENCE] Agner Fog, “Optimizing subroutines in assembly language”
• [ESTABLISHED] simdjson paper/repo — SIMD applied to a “non-numeric” problem
• [ESTABLISHED] llama.cpp ggml-quants.c — production SIMD quantization kernels
• Next: 05 — Threads, processes, context switching