A Neural Network from Scratch

The idea in one minute#

A neural network is a function built from layers. Each layer multiplies its input by a matrix of weights, adds a bias, and applies a simple non-linear function. Running the layers in order is the forward pass, and it is all that inference ever does.

Training finds the weights. Measure how wrong the output is with a loss; compute, for every weight, how the loss would change if that weight changed a little (its gradient) by applying the chain rule backwards through the layers — backpropagation; then nudge each weight a small step against its gradient. Repeat thousands of times.

This lesson builds both passes in about a hundred lines, with no library, and checks the gradients numerically so you know they are right.

An analogy#

Tuning a sound desk blindfolded. You play the track (forward pass) and someone tells you how far the result is from the target (loss). For each knob you work out whether turning it up would make things better or worse, and by how much (gradient). You turn every knob a little in its helpful direction and play the track again.

A picture#

flowchart TB
  X["x<br/>2 inputs"] --> L1["z1 = W1 x + b1<br/>16 hidden units"]
  L1 --> ACT["a1 = tanh(z1)"]
  ACT --> L2["z2 = w2 . a1 + b2"]
  L2 --> SIG["p = sigmoid(z2)"]
  SIG --> LOSS["loss = cross-entropy(p, y)"]
  LOSS -.->|"dz2 = p - y"| L2
  L2 -.->|"dw2 = dz2 * a1<br/>da1 = dz2 * w2"| ACT
  ACT -.->|"dz1 = da1 * (1 - a1^2)"| L1
  L1 -.->|"dW1 = dz1 * x"| X
  UPD["update: W = W - lr * dW"]
  LOSS --> UPD
  class X neutral
  class L1,L2 memory
  class ACT,SIG compute
  class LOSS warn
  class UPD queue

How it really works#

Forward#

For one example x with two features, a hidden layer of H units and one output:

z1 = W1·x + b1          W1 is H×2, b1 has H entries
a1 = tanh(z1)           the non-linearity: without it, two layers collapse into one matrix
z2 = w2·a1 + b2         a single number
p  = 1 / (1 + e^(−z2))  sigmoid: squashes to a probability

Loss#

For a label y that is 0 or 1, binary cross-entropy:

loss = −[ y·log(p) + (1−y)·log(1−p) ]

It is zero when the network is confident and right, and grows without bound when it is confident and wrong.

Backward#

The chain rule, applied from the loss back to each weight. With sigmoid and cross-entropy together the first step simplifies neatly:

dz2 = p − y                         how wrong the output was
dw2 = dz2 · a1        db2 = dz2
da1 = dz2 · w2                      pass the error back through the output weights
dz1 = da1 · (1 − a1²)               derivative of tanh
dW1 = dz1 ⊗ x         db1 = dz1     outer product: one gradient per weight

Every gradient has the same shape as the thing it is a gradient of. Average them over the batch.

Update#

W ← W − learning_rate · dW

Plain gradient descent. Real training uses variants (momentum, Adam) and small random batches (stochastic gradient descent), and the gradients are produced by automatic differentiation rather than by hand — but the arithmetic is this.

Checking gradients#

A hand-derived gradient is easy to get subtly wrong. The check: nudge one weight by a tiny ε, recompute the loss, and compare (loss(w+ε) − loss(w−ε)) / 2ε with your analytic gradient. They should agree to several digits. Every autodiff system is tested this way.

Training versus inference#

The last two rows are why Go fits inference: read-only weights in a flat pointer-free slice, shared by many goroutines, each with its own scratch buffers (III.05, IV.05).

What this toy leaves out#

Bigger networks differ in scale and in a few components — attention (lesson 05), normalization layers, residual connections — but not in kind. Every one is matrices, simple non-linearities, a loss, and the chain rule.

Code#

// mlp.go — a two-layer network trained with hand-written backpropagation, plus a gradient check.
package main

import (
	"fmt"
	"math"
	"math/rand"
)

const H = 16 // hidden units

type Net struct {
	W1 [H][2]float64
	B1 [H]float64
	W2 [H]float64
	B2 float64
}

type Grad = Net // a gradient has exactly the shape of the parameters

func sigmoid(z float64) float64 { return 1 / (1 + math.Exp(-z)) }

// forward returns the prediction and the hidden activations (needed by backward).
func (n *Net) forward(x [2]float64) (p float64, a1 [H]float64) {
	z2 := n.B2
	for h := 0; h < H; h++ {
		a1[h] = math.Tanh(n.W1[h][0]*x[0] + n.W1[h][1]*x[1] + n.B1[h])
		z2 += n.W2[h] * a1[h]
	}
	return sigmoid(z2), a1
}

func loss(p, y float64) float64 {
	const eps = 1e-12
	return -(y*math.Log(p+eps) + (1-y)*math.Log(1-p+eps))
}

// backward accumulates the gradient of the loss for one example into g.
func (n *Net) backward(x [2]float64, y float64, g *Grad) float64 {
	p, a1 := n.forward(x)
	dz2 := p - y
	g.B2 += dz2
	for h := 0; h < H; h++ {
		g.W2[h] += dz2 * a1[h]
		dz1 := dz2 * n.W2[h] * (1 - a1[h]*a1[h])
		g.W1[h][0] += dz1 * x[0]
		g.W1[h][1] += dz1 * x[1]
		g.B1[h] += dz1
	}
	return loss(p, y)
}

func (n *Net) step(g *Grad, lr, batch float64) {
	s := lr / batch
	n.B2 -= s * g.B2
	for h := 0; h < H; h++ {
		n.W2[h] -= s * g.W2[h]
		n.W1[h][0] -= s * g.W1[h][0]
		n.W1[h][1] -= s * g.W1[h][1]
		n.B1[h] -= s * g.B1[h]
	}
}

func main() {
	rng := rand.New(rand.NewSource(42))

	// Data: points in the square; label 1 when x and y have the same sign (the XOR pattern).
	// No straight line separates the classes, so a hidden layer is required.
	const N = 400
	xs := make([][2]float64, N)
	ys := make([]float64, N)
	for i := range xs {
		xs[i] = [2]float64{rng.Float64()*2 - 1, rng.Float64()*2 - 1}
		if xs[i][0]*xs[i][1] > 0 {
			ys[i] = 1
		}
	}

	var net Net
	for h := 0; h < H; h++ { // small random start: identical weights would stay identical
		net.W1[h] = [2]float64{rng.NormFloat64(), rng.NormFloat64()}
		net.W2[h] = rng.NormFloat64() * 0.5
	}

	// Gradient check on one weight, before training.
	var g Grad
	net.backward(xs[0], ys[0], &g)
	const eps = 1e-6
	orig := net.W1[3][1]
	net.W1[3][1] = orig + eps
	pPlus, _ := net.forward(xs[0])
	net.W1[3][1] = orig - eps
	pMinus, _ := net.forward(xs[0])
	net.W1[3][1] = orig
	numeric := (loss(pPlus, ys[0]) - loss(pMinus, ys[0])) / (2 * eps)
	fmt.Printf("gradient check on W1[3][1]: backprop %.8f, numerical %.8f\n\n", g.W1[3][1], numeric)

	// Training: full-batch gradient descent.
	fmt.Println("epoch    loss   accuracy")
	for epoch := 0; epoch <= 2000; epoch++ {
		var grad Grad
		total, correct := 0.0, 0
		for i := range xs {
			total += net.backward(xs[i], ys[i], &grad)
			if p, _ := net.forward(xs[i]); (p > 0.5) == (ys[i] == 1) {
				correct++
			}
		}
		if epoch%250 == 0 {
			fmt.Printf("%5d  %6.4f   %5.1f%%\n", epoch, total/N, 100*float64(correct)/N)
		}
		net.step(&grad, 1.0, N)
	}

	// Inference: forward only, on points the network has never seen.
	fmt.Println("\nunseen points:")
	for _, x := range [][2]float64{{0.7, 0.6}, {-0.5, -0.8}, {0.6, -0.7}, {-0.9, 0.4}} {
		p, _ := net.forward(x)
		fmt.Printf("  (%5.1f, %5.1f) → p(same sign) = %.3f\n", x[0], x[1], p)
	}
}

Remember this#

• Forward: matrix multiply, add, non-linearity, repeat. Inference is only this.
• Training: loss, gradients by the chain rule run backwards, a small step against each gradient.
• A gradient has the shape of its parameter. Verify hand-written gradients numerically.
• Inference weights are read-only, which makes them safely shareable across goroutines.

Try it#

1. Run mlp.go. Set H to 2. Can the network still learn the pattern? Why does width matter here?
2. Remove the tanh (use a1 = z1). What accuracy do you get, and what does that say about non-linearity?
3. Switch to mini-batches of 32 shuffled examples per step. Does the loss fall faster per example processed?

Check yourself#

1. What does the backward pass compute?
2. Why must training keep every layer’s activations while inference need not?
3. How do you verify a hand-derived gradient?