PromQL Essentials

The idea in one minute#

PromQL is the query language of Prometheus and of most systems compatible with it. Nearly every useful query is the same four steps: select series by name and labels, turn counters into per-second rates over a time window, aggregate across the labels you do not care about, and combine two results with arithmetic.

Learn rate, sum by, division and histogram_quantile, and you can read 90% of the dashboards and alert rules you will ever meet.

An analogy#

A spreadsheet pivot table. Select the rows you want (filter), convert running totals to per-day figures (rate), collapse the columns you do not care about (sum by region), and divide one column by another (errors ÷ total).

A picture#

flowchart LR
  S["Select<br/>http_requests_total{job='api'}"] --> R["Rate<br/>rate(...[5m])<br/>per-second, per series"]
  R --> A["Aggregate<br/>sum by (route) (...)"]
  A --> C["Combine<br/>errors / total"]
  C --> OUT["One number per route:<br/>error ratio"]
  class S neutral
  class R compute
  class A queue
  class C compute
  class OUT memory

How it really works#

Selecting#

http_requests_total                          # every series with this name
http_requests_total{route="/v1/chat"}        # exact match
http_requests_total{code=~"5.."}             # regular expression
http_requests_total{code!~"2..|3.."}         # negative regular expression
http_requests_total[5m]                      # a range: the last 5 minutes of samples

A selector without [...] gives an instant vector (one value per series, now). With [5m] it gives a range vector (a window of samples per series), which is what rate consumes.

rate: counters into per-second values#

rate(http_requests_total[5m])

For each series: (increase over the window) ÷ (seconds in the window), correcting for counter resets. When a process restarts and its counter drops from 9,000 to 12, rate treats the drop as a reset and adds 12, not −8,988.

Rules of thumb:

• The window should cover at least four scrape intervals ([1m] for a 15 s scrape). In Grafana use $__rate_interval, which picks this for you.
• rate for alerts and graphs; irate (last two samples only) only for zoomed-in graphs; increase(x[1h]) is rate × 3600, for “how many in the last hour”.
• rate first, then sum. sum first destroys the per-series resets and produces garbage when any instance restarts.

Aggregating#

sum by (route) (rate(http_requests_total[5m]))        # keep only `route`
sum without (instance, pod) (rate(...[5m]))           # drop these, keep the rest
max by (model) (queue_depth)                          # worst replica per model
topk(5, sum by (tenant) (rate(tokens_total[5m])))     # the five busiest tenants

Operators: sum, avg, min, max, count, topk, bottomk, quantile.

Combining: ratios#

# Error ratio per route
  sum by (route) (rate(http_requests_total{code=~"5.."}[5m]))
/
  sum by (route) (rate(http_requests_total[5m]))

Binary operators match series whose labels are identical on both sides. If one side has extra labels, say so: ... / on (route) group_left .... Always divide sums, never average ratios: the mean of per-instance error ratios weights an idle instance the same as a busy one.

Percentiles from histograms#

# Classic histogram: keep the `le` label when aggregating
histogram_quantile(0.99,
  sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))

# Native histogram: no _bucket suffix, no le label
histogram_quantile(0.99,
  sum by (route) (rate(http_request_duration_seconds[5m])))

# Average duration (useful as a sanity check, not as an SLI)
  rate(http_request_duration_seconds_sum[5m])
/ rate(http_request_duration_seconds_count[5m])

# Fraction of requests faster than 250 ms — a latency SLI, exact if 0.25 is a bucket bound
  sum(rate(http_request_duration_seconds_bucket{le="0.25"}[5m]))
/ sum(rate(http_request_duration_seconds_count[5m]))

Lesson 04 explains what histogram_quantile is doing and how far off it can be.

Gauges over time#

avg_over_time(queue_depth[10m])     max_over_time(gpu_temperature_celsius[1h])
predict_linear(disk_free_bytes[6h], 4 * 3600) < 0    # will it be full in four hours?

Absence#

A query for a missing series returns nothing, and an alert on “nothing” never fires. Alert on absence explicitly: up{job="api"} == 0 for a failed scrape, absent(up{job="api"}) when the target disappeared altogether.

Recording rules#

A query that is expensive or used in many places can be evaluated on a schedule and saved as a new series:

groups:
  - name: api
    rules:
      - record: route:http_requests:rate5m
        expr: sum by (route) (rate(http_requests_total[5m]))

The naming convention is level:metric:operations. Dashboards and alerts then read the pre-computed series.

Code#

What rate() does, including reset handling — the detail that makes counters safe.

// rate.go — PromQL's rate() over a window of counter samples, with reset correction.
package main

import "fmt"

type Sample struct {
	T float64 // seconds
	V float64
}

// rate returns the per-second increase over the samples, treating any decrease as a reset.
func rate(s []Sample) float64 {
	if len(s) < 2 {
		return 0
	}
	increase := 0.0
	for i := 1; i < len(s); i++ {
		d := s[i].V - s[i-1].V
		if d < 0 { // counter reset: the process restarted and began again from zero
			d = s[i].V
		}
		increase += d
	}
	return increase / (s[len(s)-1].T - s[0].T)
}

func naive(s []Sample) float64 {
	return (s[len(s)-1].V - s[0].V) / (s[len(s)-1].T - s[0].T)
}

func main() {
	steady := []Sample{{0, 1000}, {15, 1150}, {30, 1300}, {45, 1450}, {60, 1600}}
	restart := []Sample{{0, 9000}, {15, 9150}, {30, 12}, {45, 162}, {60, 312}}

	fmt.Printf("steady counter:        rate = %6.2f/s   naive = %7.2f/s\n", rate(steady), naive(steady))
	fmt.Printf("restart in the window: rate = %6.2f/s   naive = %7.2f/s\n", rate(restart), naive(restart))

	// Why rate-then-sum: summing first hides the reset inside a larger number.
	a := []Sample{{0, 5000}, {15, 5150}, {30, 5300}, {45, 5450}, {60, 5600}}
	summed := make([]Sample, len(a))
	for i := range a {
		summed[i] = Sample{a[i].T, a[i].V + restart[i].V}
	}
	fmt.Printf("\ntwo instances, one restarts:\n")
	fmt.Printf("  sum(rate(x)) = %6.2f/s   (correct)\n", rate(a)+rate(restart))
	fmt.Printf("  rate(sum(x)) = %6.2f/s   (wrong: one instance's restart is treated as a reset of the whole sum)\n", rate(summed))
}

Remember this#

• Select → rate → aggregate → combine.
• rate before sum, always. Windows of at least four scrape intervals.
• Divide sums to get ratios; never average ratios or percentiles.
• Keep le when aggregating a classic histogram for histogram_quantile.
• Alert on absence explicitly.

Try it#

1. Run rate.go. Make the restart happen between two scrapes where the counter had grown by 140 before dying. How much does rate under-count, and why is that unavoidable?
2. Write PromQL for: requests per second by status class; the error ratio for one route; the p95 duration across all routes.
3. Start Prometheus against lesson 02’s server and try your queries. Then restart the server and watch rate stay sane.

Check yourself#

1. What is the difference between an instant vector and a range vector?
2. Why must rate come before sum?
3. Why is “average of per-instance error ratios” wrong?