Observability & Distributed Tracing
Implement observability with OpenTelemetry: distributed tracing, structured logs, metrics, and correlation IDs across microservices.
Observability is the ability to understand a system's internal state from its external outputs (logs, metrics, traces). In distributed systems, traditional logging fails - requests span multiple services, making debugging without distributed tracing like navigating with no GPS.
Use OpenTelemetry (OTel) as your single observability framework - vendor-neutral, with auto-instrumentation for most frameworks. The three pillars: traces (what happened, in what order), metrics (how many, how fast, how much), logs (what the system said). Correlate all three with trace IDs.
Workflow
- Set up OpenTelemetry auto-instrumentationAuto-instrumentation captures HTTP, database, and framework spans without code changes - 80% of value for 20% of effort.
- Create custom spans for business operationsAuto-instrumentation misses business logic context; custom spans add meaning to traces.
- Implement structured logging with trace correlationCorrelating logs with traces lets you jump from a slow span to the relevant log lines.
- Define RED metrics per serviceRED (Rate, Errors, Duration) is the standard metric framework for service health.
- Route telemetry through OpenTelemetry CollectorThe OTel Collector decouples your app from the backend - swap Jaeger for Tempo without app changes.
Output Format
OTel auto-instrumentation, custom spans for business logic, structured logs with traceId, RED metrics with alerts, OTel Collector routing to backends.
Examples
Custom span: `await tracer.startActiveSpan("sendEmail", async (span) => { span.setAttributes({ "email.to": to, "email.template": template }); try { await sendgrid.send(msg); span.setStatus({ code: SpanStatusCode.OK }); } catch (e) { span.recordException(e); span.setStatus({ code: SpanStatusCode.ERROR }); } finally { span.end(); } })`Structured log: `logger.info({ traceId: trace.getActiveSpan()?.spanContext().traceId, event: "order_created", orderId, userId, total }, "Order created successfully")`Metric: `const latencyHistogram = meter.createHistogram("payment.processing.duration", { unit: "ms" }); const start = Date.now(); await processPayment(); latencyHistogram.record(Date.now() - start, { provider: "stripe" })`Edge Cases & Gotchas
- Trace sampling: 100% sampling in high-traffic production is expensive; use tail-based sampling (sample slow/error requests) or head-based sampling at 5-10% for normal traffic
- Baggage propagation: for multi-service traces, use OTel Baggage to propagate business context (userId, tenantId) across service boundaries via HTTP headers
- Clock skew: distributed services have slightly different clocks; Jaeger and Tempo account for this in trace visualization, but extreme skew (>1s) can mis-order spans
When to Break These Rules
- For single-service apps with <100 RPS, structured logging + basic Prometheus metrics is sufficient - distributed tracing's complexity is only justified at scale or with multiple services.
Reference Files
OpenTelemetry: https://opentelemetry.io/docs/Grafana LGTM stack: https://grafana.com/oss/Jaeger: https://www.jaegertracing.io/Google SRE RED method: https://www.weave.works/blog/the-red-method-key-metrics-for-microservices-architecture/