Observability: Logs, Metrics and Traces

For a single server, monitoring meant a few graphs (CPU, memory, disk) and tail -f on a log file. Splitting systems into dozens of services broke that: a slow checkout could be caused by any of them, and no single log file held the whole story. Google described its answer, request tracing, in the Dapper paper (2010); Twitter open-sourced Zipkin in 2012 and Uber released Jaeger in 2017. Prometheus (SoundCloud, 2012; the second project in the CNCF, the Cloud Native Computing Foundation, in 2016) made pull-based metrics the default in Kubernetes clusters. The word observability, borrowed from control theory, spread from around 2016 to mean being able to ask new questions of a system without shipping new code. In 2019 the two competing open tracing standards, OpenTracing and OpenCensus, merged into OpenTelemetry, which now covers all three signals.

You have met all three already: a JSON line in your terminal from a logger such as pino, a Grafana dashboard with request rate and latency, and the waterfall view in Jaeger or Datadog that shows which call was slow. The smallest useful example is one log line that carries the ID of the request it belongs to:

Incidents are measured in time to detect and time to resolve, and both depend on signals that existed before the incident started. Without metrics you learn about an outage from users. Without traces you know checkout is slow but not which of eight services is slow. Without correlated logs you find the trace but not the error that explains it. The opposite failure is just as common: a logging bill larger than the compute bill, a metrics database brought down by one label, and alerts so noisy that the team mutes the channel and misses the real page.

The signals differ in what they keep. A log line keeps every detail of one event. A metric throws the details away and keeps a running number per time window, which is why it is cheap to store for months and fast to graph. A trace keeps the timing and causal structure of one request across process boundaries, which no other signal can reconstruct.

A log line like Payment failed for order 8812 can only be grepped. The same event as key-value fields can be filtered, grouped and joined: every line from service=payments with http.status=402 in the last hour, or every line of one request. That last query only works if each line carries a correlation ID, and the best one is the trace ID, because the tracing system already propagates it across services for free.

Most metrics are one of three types. A counter only goes up (requests served, errors), and you graph its rate. A gauge goes up and down (memory in use, queue length). A histogram counts observations into buckets (requests under 100 ms, under 250 ms, …), which is what lets you compute percentiles across many instances. You cannot average percentiles from different servers, but you can add their bucket counts and take the percentile of the sum.

A time-series database (TSDB) such as Prometheus stores one series for every distinct combination of label values. Labels like route, method and status are fine, because they multiply to a few hundred series. Add user_id and every user becomes a new series per route per status: memory grows with users, queries slow down, and the database eventually falls over. Unbounded values belong in logs and trace attributes, never in metric labels.

Three checklists cover almost every service. The four golden signals from the Google SRE book (2016) are latency, traffic, errors and saturation. RED (rate, errors, duration), named by Tom Wilkie, is the same idea for request-driven services. USE (utilization, saturation, errors), from Brendan Gregg, applies to resources such as CPU, disks, connection pools and queues. RED tells you users are affected; USE tells you which resource ran out.

When a request enters the system, the first service creates a trace ID and a root span. Every outgoing call carries the trace ID and the current span ID in a header; the receiving service creates a child span with that parent and does the same for its own calls. Each service sends its finished spans to a collector independently, and the backend reassembles them by trace ID into a tree with timings.

The header format was standardised as W3C Trace Context (a W3C Recommendation since 2020), so services written in different languages and traced by different vendors still join one trace. The traceparent header holds a version, the 128-bit trace ID, the caller's span ID and a flags byte whose last bit says whether this trace is being recorded:

Recording every span of every request is expensive at scale, so traces are sampled. Head sampling decides at the first service (keep 5% of traces) and passes the decision downstream in the flags; it is cheap but blind, so most of the rare slow or failed requests are thrown away. Tail sampling buffers all spans of a trace in a collector and decides after it finishes: keep every error, every request over 1 second and 1% of the rest. It keeps the interesting traces at the price of running and sizing that buffer.

OpenTelemetry (OTel) provides one API and SDK per language for traces, metrics and logs, auto-instrumentation for common HTTP clients, servers and database drivers, and OTLP (the OpenTelemetry Protocol) for shipping data. Its tracing API reached 1.0 in 2021, and metrics and logs followed as stable specifications. Because the API is vendor-neutral, switching from Jaeger to Datadog or Grafana Tempo is a collector configuration change, not a code change.

An alert should mean a human must act now. Alerts on causes (CPU at 90%, a pod restarted) fire constantly without user impact, and miss failures nobody thought to write a rule for. Alerts on symptoms catch both: the share of requests that are slow or failing. An SLO makes that precise and gives the team an error budget, the amount of unreliability it can spend on deploys and experiments before reliability work takes priority.