Observability Engineering

Learn to see inside running systems: what to measure, how the measurements are collected and stored, how to turn them into decisions — and what changes when the system is a rack of GPUs generating tokens.

This course starts at “what is a metric?” and ends at “design the telemetry for a multi-tenant inference platform”. It assumes you can program in Go. It does not assume you have run Prometheus, read a trace, or been on call.

It leans on two other paths for the systems being observed — Inference Engineering and GPU Engineering — and on Golang Engineering if you want the language itself in depth. You can take modules I–IV without any of them.

How this course works#

Every lesson follows the same shape:

1. The idea in one minute — the whole lesson in a few sentences.
2. An analogy — something from everyday life with the same shape.
3. A picture — a hand-drawn diagram of the mechanism.
4. How it really works — the precise version, with real names and numbers.
5. Code — a small Go program you can run. Almost all of them use only the standard library.
6. Remember this — the three or four facts worth keeping.
7. Try it and Check yourself — exercises and questions.

Why build the tools before using them?#

You will not use a hand-written metrics library in production; you will use Prometheus client libraries and OpenTelemetry. But a counter is twenty lines of Go, a histogram is forty, and a trace context is one HTTP header. Building each one once removes the mystery, and the mystery is what makes people instrument badly: labels with unbounded values, averages of percentiles, alerts on causes instead of symptoms. Each lesson builds the mechanism first and then shows the production tool that does the same job.

flowchart LR
  APP["Your service<br/>counters, spans, logs"] --> COL["Collection<br/>scrape or push"]
  COL --> STORE[("Storage<br/>time series, columns, objects")]
  STORE --> Q["Query<br/>PromQL, SQL, trace search"]
  Q --> DASH["Dashboards"]
  Q --> ALERT["Alerts"]
  ALERT --> HUMAN["A person<br/>or an agent"]
  HUMAN -->|"asks a new question"| Q
  class APP compute
  class COL io
  class STORE memory
  class Q,ALERT queue
  class DASH,HUMAN neutral

The path#

flowchart LR
  I["I Why observability"] --> II["II Metrics"]
  II --> III["III Logs, traces, profiles"]
  III --> IV["IV Running it"]
  IV --> V["V AI infrastructure"]
  V --> VI["VI Tools and frontier"]
  class I neutral
  class II,III compute
  class IV queue
  class V memory
  class VI io

What you need#

• Go 1.22 or newer and a terminal.
• Nothing else for modules I–III: every program runs offline.
• Docker (or Podman) for the optional exercises that start Prometheus, Grafana or an OpenTelemetry Collector.
• No GPU required. Module V explains GPU and engine telemetry with simulators and real metric names; the exercises that read a real GPU say so and give a no-GPU alternative.

Getting started in fifteen minutes#

1. Read I.01 and run its program.
2. Read II.01 and II.02; curl your own /metrics endpoint.
3. If you came for AI infrastructure and already know Prometheus, jump to V.01 and come back to module II’s histograms and cardinality lessons when a percentile or a bill surprises you.

A promise about names and versions#

Metric names, tool versions and project statuses in this course were checked on 3 October 2026. Mechanisms — what a counter is, why a percentile cannot be averaged, why a GPU-utilization gauge misleads — do not change. Names do: a lesson says so wherever a name comes from a specification that is still marked unstable, and VI.03 lists where to check what has moved.