What to Watch

The idea in one minute#

Two lists. First, the metrics to watch on an AI serving platform — about two dozen, layer by layer, each with the direction that means trouble. Second, the projects and feeds to watch, with their status on 3 October 2026 and where to re-check it.

Use the first list as a checklist against your own dashboards. Use the second on a schedule: once a quarter, re-read the statuses and update what you depend on.

A picture#

flowchart LR
  U["User experience<br/>TTFT, TPOT, errors, goodput"] --> E["Engine<br/>waiting, KV usage, preemptions, cache hits"]
  E --> H["Hardware<br/>power, throttling, errors, links"]
  E --> P["Platform<br/>pending, cold start, outliers, routing"]
  P --> C["Cost<br/>$ per M tokens, occupancy, tokens per joule"]
  U --> Q["Quality<br/>finish reasons, parse failures, eval scores"]
  class U queue
  class E memory
  class H io
  class P compute
  class C neutral
  class Q queue

How it really works#

The metrics to watch#

Start from the top: only descend when a row above is unhealthy.

User experience — alert on these

Engine — the first place to look

Hardware — is the device healthy and working

Platform — is capacity arriving and being shared fairly

Cost and efficiency — review weekly

Quality — gate rollouts on these

Projects and standards: status on 3 October 2026#

Feeds worth a regular look#

Vendor blogs report their best case. Read them for mechanisms, and verify numbers yourself.

A quarterly review, in six questions#

1. Did any metric or attribute we alert on get renamed upstream?
2. Is anything we depend on still below “stable”? Did it move?
3. Which dashboards did nobody open? Delete them.
4. Which pages needed no action? Fix or delete them.
5. What did the last incident lack? Was it added?
6. What did telemetry cost, per team, against the infrastructure it watches?

Evaluating a new tool or signal#

Ask, in order: which job does it do (lesson 01)? Does it speak OTLP / PromQL, or its own format? What is its stability label? Can it be joined to what you have — same resource attributes? What does it cost at your volume? What is the exit?

Code#

Turn the first list into something executable: a health roll-up that names the first unhealthy layer, which is where an investigation should start.

// watch.go — evaluate a snapshot of the key metrics and point at the first unhealthy layer.
package main

import "fmt"

type Check struct {
	Layer, Name string
	Value       float64
	Bad         func(float64) bool
	Unit        string
}

func main() {
	gt := func(limit float64) func(float64) bool { return func(v float64) bool { return v > limit } }
	lt := func(limit float64) func(float64) bool { return func(v float64) bool { return v < limit } }

	// A snapshot. In production each value is the result of one query.
	checks := []Check{
		{"1 user", "TTFT p95", 0.92, gt(0.5), "s"},
		{"1 user", "TPOT p95", 0.031, gt(0.040), "s"},
		{"1 user", "goodput", 0.91, lt(0.99), "ratio"},
		{"2 engine", "requests waiting (max replica)", 2, gt(8), ""},
		{"2 engine", "KV-cache usage", 0.71, gt(0.90), "ratio"},
		{"2 engine", "preemptions per min", 0, gt(0), ""},
		{"2 engine", "prefix-cache hit rate", 0.22, lt(0.50), "ratio"},
		{"3 hardware", "throttling seconds per min", 0, gt(0), "s"},
		{"3 hardware", "hardware Xid events (1 h)", 0, gt(0), ""},
		{"4 platform", "pods pending for a GPU", 0, gt(0), ""},
		{"4 platform", "slowest replica TTFT vs fleet", 1.2, gt(2), "x"},
		{"5 quality", "finish_reason=length share", 0.012, gt(0.03), "ratio"},
	}

	first := ""
	for _, c := range checks {
		status := "ok"
		if c.Bad(c.Value) {
			status = "BAD"
			if first == "" || (c.Layer > "1 user" && first == "1 user") {
				first = c.Layer
			}
		}
		fmt.Printf("%-11s %-34s %8.3f %-6s %s\n", c.Layer, c.Name, c.Value, c.Unit, status)
	}
	fmt.Printf("\nStart investigating at: %s\n", first)
	fmt.Println("Here: users see slow first tokens, the queue is short, the hardware is healthy,")
	fmt.Println("and the prefix cache stopped hitting — look for a prompt or routing change, not for capacity.")
}

Remember this#

• About two dozen metrics cover an AI serving platform; read them top-down from user experience.
• Each has a direction that means trouble; write it next to the panel.
• Standards and tools carry stability labels. Re-check them quarterly, from primary sources.
• Evaluate anything new by job, standard, stability, joinability, cost and exit.

Try it#

1. Run watch.go. Change the snapshot so the cause is hardware throttling. Which values move together?
2. Tick off the metric tables against your own dashboards. What is missing from each layer?
3. Do the six-question quarterly review for a system you know.

Check yourself#

1. In what order do you read the layers during an incident, and why?
2. Which three items in the status table were not yet stable in October 2026?
3. What six questions do you ask of a new tool?