Insights · Platform Engineering

Observability that answers questions: logs, metrics and traces in production

PrometheusGrafanaSplunk

Key takeaway

Logs, metrics, traces: instrument once with OpenTelemetry, drive decisions with SLOs, and alert only on symptoms your users feel.

Monitoring asks predefined questions; observability is the property that lets you ask ones you didn’t foresee. The OpenTelemetry primer defines it operationally: “understand a system from the outside by letting you ask questions” about a system you can’t open and inspect. The distinction matters most in distributed estates, where the failure you’re debugging at 3 a.m. is, by definition, one nobody wrote a dashboard for — the “unknown unknowns” argument Charity Majors makes on the Bike Shed podcast: real-time questions of high-cardinality data, without re-instrumenting the code.

The three signals, and what each is for

The signals are complementary instruments, not a checklist of products:

  • Logs — “a timestamped message emitted by services”: discrete, structured events. The workhorse of forensic reconstruction. Make them structured (JSON with stable fields) or they are prose, not data.
  • Metrics — “aggregations over a period of time of numeric data”: cheap, alertable, storage-friendly. The health layer for infrastructure, and the substrate for SLOs. Their weakness is baked in: aggregation destroys the individual request, and with it the long tail.
  • Traces — “the path taken by a single request”: spans linked across services, queues and databases. Google’s Dapper paper established the recipe that made this viable at scale — library-level instrumentation with negligible overhead, plus sampling. A 2025 quantitative analysis of production tracing data from an Alibaba microservice deployment (arXiv) shows why traces are irreplaceable at scale: extreme call-graph variability and long-tail workloads that aggregate metrics structurally cannot represent.

The practitioner synthesis: metrics tell you that something is wrong, traces tell you where, logs tell you why. Systems that try to make one signal do all three jobs pay more and learn less.

The four golden signals

For what to measure first, the Google SRE book’s monitoring chapter remains the canonical shortlist — the four golden signals of user-facing service health:

  • Latency — how long requests take, with the book’s sharper point that “a slow error is even worse than a fast error”: failed-but-slow calls poison the latency chart unless errors are separated.
  • Traffic — demand on the service, per protocol (requests/second, transactions, concurrent sessions).
  • Errors — the rate of failed requests, explicit and implicit.
  • Saturation — “how ‘full’ your service is”, against the constraint that will exhaust first (connections, threads, queue depth).

The SRE chapter’s deeper instruction is to “page on symptoms” — what users are experiencing — rather than causes, which is the discipline that separates actionable alerts from noise.

SLOs and error budgets: from signals to decisions

Signals become architecture when they’re bound to objectives. An SLO is a target on a service-level indicator — say, 99.9% of requests under 300 ms over 30 days — and the SRE book is explicit about the philosophy: “it’s both unrealistic and undesirable to insist that SLOs will be met 100% of the time.” The gap between target and reality is the error budget: the SRE Workbook’s alerting chapter does the math plainly — “the error budget gives the number of allowed bad events” (99.9% over 30 days leaves 43 minutes of full downtime) — and defines burn rate as “how fast, relative to the SLO, the service consumes the error budget.” Budget burning fast: halt feature deploys and stabilize. Budget intact: ship. The budget converts reliability from a feeling into a currency that funds velocity — and it gives the event backbone and the platform teams a shared, arguable number.

Instrument once, propagate everywhere

The instrumentation layer consolidated for good reason. OpenTelemetry — “a vendor-neutral open source” framework covering “traces, metrics, logs” as its signals, now CNCF-graduated — means you instrument applications once and route telemetry to whichever backend survives your procurement cycle. The property that makes traces work across a distributed estate is context propagation: the W3C Trace Context recommendation standardizes the traceparent and tracestate headers “to propagate context information that enables distributed tracing scenarios” — which is how one trace spans the API gateway, five services, and the message queue between them. Propagation through asynchronous boundaries (event headers carrying trace context) is the difference between a distributed system you can debug and one you can only restart.

Research backs treating observability configuration as a designed artifact: Borges et al. (ICSA 2024) observe that observability setups are usually “intuition-guided” and tool-dependent, and propose fault-injection experiments that test whether your telemetry actually catches the failure modes you care about — observability with a test suite, not a vibe.

Sampling: the cost lever

Telemetry at scale is a storage bill, and sampling is how you pay it deliberately. The research shows the lever is long: a 2025 span-level sampling approach (Wu et al., arXiv) cut trace volume by 81.2% while retaining 98.1% coverage of faulty spans — and improved root-cause analysis. The directional rules: keep 100% of error traces (failures are rare and precious), head-sample or tail-sample the healthy traffic, and set cardinality budgets on metrics (unbounded labels are a self-inflicted outage of your own observability stack).

What good looks like in production

  1. Every service ships OTel-instrumented traces with context propagated across every hop, including the backbone.
  2. Dashboards and alerts start from the golden signals, paging on user-felt symptoms with burn-rate alerts on SLOs.
  3. Logs are structured, correlated by trace ID, and queryable alongside traces.
  4. Error traces are never sampled away; healthy-trace sampling is tuned against the bill.
  5. Every incident ends by asking one question: which novel question do we now need to answer in production — and can the current telemetry answer it?

Instrument for the questions you can’t predict

Sources