Histograms

The idea in one minute#

A histogram does not store measurements; it stores how many fell into each bucket. A percentile computed from it is an estimate: the query finds the bucket containing the target rank and assumes values are spread evenly inside it. The error is therefore at most one bucket width — so bucket boundaries decide your accuracy.

Classic histograms make you choose boundaries in advance, and you will choose wrong. Native histograms (stable in Prometheus since 3.8) use automatically-placed exponential buckets: one series instead of a dozen, and a bounded relative error everywhere.

An analogy#

Estimating people’s heights from a tally sheet with rows “under 150 cm”, “150–200 cm”, “over 200 cm”. You can say the median is “somewhere between 150 and 200”. If the rows were 5 cm apart you could say something useful. The tally sheet is the histogram; the row spacing is your resolution.

A picture#

flowchart TB
  OBS["Observation: 0.37 s"] --> FIND["Find its bucket"]
  FIND --> B["bucket (0.25, 0.5] += 1<br/>sum += 0.37, count += 1"]
  B --> SCRAPE["Scraped as cumulative counters"]
  SCRAPE --> Q["histogram_quantile(0.99, ...)"]
  Q --> RANK["Target rank = 0.99 x count"]
  RANK --> WHICH["Which bucket holds that rank?"]
  WHICH --> INTERP["Interpolate inside it<br/>assumes an even spread"]
  INTERP --> EST["Estimate, wrong by up to<br/>one bucket width"]
  class OBS neutral
  class FIND,RANK,WHICH queue
  class B,SCRAPE memory
  class Q,INTERP compute
  class EST warn

How it really works#

Classic histograms#

A classic histogram with bounds 0.1, 0.25, 0.5, 1 exposes one counter per bound (cumulative, labelled le), plus +Inf, _sum and _count. Consequences:

• Cost: (bounds + 3) series per label combination. Twelve buckets × 50 routes × 20 instances = 15,000 series for one metric.
• Accuracy: fine where buckets are dense, poor where they are sparse. If your SLO threshold is 300 ms and your bounds are 250 and 500, p99 = “somewhere in there”.
• Aggregation: only histograms with identical bounds can be summed.
• The top bucket: if the target rank falls in +Inf, the estimate is simply the highest finite bound. A p99 that sits exactly on your top bound means “off the chart”.

Choosing bounds, if you must: put a bound exactly at each SLO threshold (the fraction-below query from lesson 03 is then exact), space the rest roughly geometrically (each ~2× the last), and cover from below your fastest request to above your timeout.

Native histograms#

A native histogram is one series whose samples are whole histograms. Buckets are exponential: each boundary is the previous one times a fixed factor, 2^(2^-schema).

Only buckets that have received observations are stored, so the cost follows the spread of the data, not the number of possible buckets. The library lowers the resolution automatically if a bucket limit is exceeded. Because every native histogram uses the same family of boundaries, any two can be merged — a finer one is downscaled to match a coarser one.

OpenTelemetry’s exponential histogram is the same design, and Prometheus converts OTLP exponential histograms into native histograms on ingestion.

Migration path: have the client expose both for a while, switch dashboards and alerts to the native form, then drop the classic buckets.

Reading a heatmap#

A histogram over time is best drawn as a heatmap: time on the x-axis, latency on the y-axis, colour for count. It shows what a p99 line cannot — two distinct modes (a fast path and a slow path), or a band of requests sitting exactly at a timeout.

What histograms still cannot do#

• Tell you which requests were slow. Attach exemplars — a trace ID stored alongside a bucket — and jump from the spike to a trace (III.02).
• Give an exact maximum.
• Survive bad label choices: a histogram multiplies cardinality by its bucket count (lesson 05).

Code#

How wrong is a percentile from poorly-chosen buckets, compared with exponential ones?

// buckets.go — percentile error: hand-picked buckets vs exponential (native-style) buckets.
package main

import (
	"fmt"
	"math"
	"math/rand"
	"sort"
)

// quantile estimates q from cumulative bucket counts, as histogram_quantile does.
func quantile(q float64, bounds []float64, counts []int) float64 {
	total := 0
	for _, c := range counts {
		total += c
	}
	rank := q * float64(total)
	cum := 0.0
	for i, c := range counts {
		if cum+float64(c) >= rank {
			if i >= len(bounds) { // the +Inf bucket: nothing to interpolate towards
				return bounds[len(bounds)-1]
			}
			lo := 0.0
			if i > 0 {
				lo = bounds[i-1]
			}
			return lo + (bounds[i]-lo)*(rank-cum)/float64(c)
		}
		cum += float64(c)
	}
	return math.NaN()
}

func fill(bounds, xs []float64) []int {
	counts := make([]int, len(bounds)+1)
	for _, x := range xs {
		counts[sort.SearchFloat64s(bounds, x)]++
	}
	return counts
}

func expBounds(lo, hi, factor float64) []float64 {
	var b []float64
	for v := lo; v < hi*factor; v *= factor {
		b = append(b, v)
	}
	return b
}

func main() {
	rng := rand.New(rand.NewSource(11))
	xs := make([]float64, 200000)
	for i := range xs {
		xs[i] = 0.08 * math.Exp(rng.NormFloat64()*0.9) // log-normal: median 80 ms, long tail
	}
	sorted := append([]float64{}, xs...)
	sort.Float64s(sorted)
	exact := func(q float64) float64 { return sorted[int(q*float64(len(sorted)))-1] }

	sets := []struct {
		name   string
		bounds []float64
	}{
		{"default-ish classic (11)", []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10}},
		{"sparse classic (4)", []float64{.1, .5, 1, 5}},
		{"exponential x2 (schema 0)", expBounds(0.001, 20, 2)},
		{"exponential x1.09 (schema 3)", expBounds(0.001, 20, math.Pow(2, 1.0/8))},
	}
	fmt.Println("buckets                        n    p50 err   p90 err   p99 err")
	for _, s := range sets {
		c := fill(s.bounds, xs)
		fmt.Printf("%-28s %3d", s.name, len(s.bounds))
		for _, q := range []float64{0.5, 0.9, 0.99} {
			est, ex := quantile(q, s.bounds, c), exact(q)
			fmt.Printf("  %+7.1f%%", 100*(est-ex)/ex)
		}
		fmt.Println()
	}
	fmt.Printf("\nexact: p50=%.0f ms  p90=%.0f ms  p99=%.0f ms\n", exact(.5)*1000, exact(.9)*1000, exact(.99)*1000)
}

Remember this#

• A histogram percentile is an estimate, accurate to about one bucket width.
• Put a classic bucket boundary exactly on each SLO threshold.
• Native (exponential) histograms: one series, automatic buckets, bounded error, always mergeable. Prefer them where your backend supports them.
• Use heatmaps to see shape, exemplars to reach the actual slow request.

Try it#

1. Run buckets.go. Move the distribution’s median to 2 s. Which bucket sets fall apart?
2. Add a bound at exactly your imagined SLO threshold and compute the fraction of requests below it from the buckets. Compare with the exact fraction.
3. Count the series each bucket set costs for 50 routes × 20 instances.

Check yourself#

1. Why is a histogram percentile only an estimate?
2. What happens when the target rank lands in the +Inf bucket?
3. Give two advantages of native histograms over classic ones.