Logs, metrics and traces: what each one tells you

Metrics tell you that something is wrong and how much: numbers such as request rate, error rate and latency, aggregated over time. Traces show where, by following one request through your services and timing each step. Logs explain why, with the detail of single events such as an error message and the input that caused it.

Observability is how well you can explain a system's behavior from the data it emits. The three signals overlap, but each is the cheapest way to answer one kind of question.

A metric is a named numeric measurement recorded at regular intervals, usually with labels such as service, route or status code. Counters only go up, like total requests served. Gauges go up and down, like memory in use. Histograms count values in buckets, from which you can compute latency percentiles.

Because a metric stores aggregates instead of individual events, it is cheap to keep for months and fast to query. That makes metrics the usual basis for dashboards and alerts. The cost is detail: a metric can show that errors rose but not which request failed.

Google's SRE book suggests starting with four signals for a user-facing system: latency, traffic, errors and saturation.

A trace records one request as a tree of spans. Each span is one unit of work, such as an HTTP handler, a database query or a call to another service. Every span has a start time, a duration and attributes.

All spans in a trace share a trace ID, and each span records its parent. Services pass this trace context along with every outgoing call, commonly in the W3C traceparent header, so the next service adds its spans to the same trace.

A trace shows where the time went and which downstream call failed. Traces are most useful in systems where one request touches several services or queues.

A log is a timestamped record of one event. Structured logs write each event as named fields, often JSON, instead of free text, so you can filter by route, user or error code without regular expressions.

Logs carry the most detail per event: error messages, stack traces and the values that failed validation. Keep passwords, tokens and personal data out of them, since logs are copied, shipped and kept for long periods.

The signals are most useful when you can move from one to the next. Write the current trace ID and span ID into every log line, as in the example above, so a slow span leads straight to the logs it produced.

Use the same service and route names in metric labels, span attributes and log fields. Some metric systems also store exemplars, which attach a sample trace ID to a histogram bucket and link a latency spike to a real request. OpenTelemetry provides one set of APIs and naming conventions for all three signals.

A typical investigation follows that path. An alert fires on the error-rate metric, traces from that window show which service failed, and that span's logs give the error message.

Each signal grows expensive in a different way, so each needs its own controls.