Monitoring & Observability
Understand your system's behavior through metrics, logs, and traces - detecting issues before users report them and diagnosing root causes in minutes.
skill: Monitoring & Observability category: DevOps tags: [Observability, Prometheus, Grafana, OpenTelemetry, Logging] readingTime: 7
Observability is not optional for production systems. Without it, you're flying blind - outages go undetected for hours and RCA takes days. Instrument from day one: structured logs cost nothing, metrics add minimal overhead, and traces pay for themselves the first time you debug a distributed latency issue.
Workflow
- Emit structured logsJSON logs are machine-parseable - log aggregators (Loki, Cloudwatch, Datadog) can filter, group, and alert on structured fields. Plain text logs require fragile regex parsing.
- Expose metrics with PrometheusPrometheus scrapes /metrics endpoints on a schedule and stores time-series data. It's the standard for metrics in Kubernetes and cloud-native environments.
- Visualize with Grafana dashboardsGrafana turns raw Prometheus metrics into actionable dashboards - request rate, error rate, latency (RED method), and infrastructure utilization.
- Add distributed tracing with OpenTelemetryIn microservices, a single user request spans many services. Traces show the full request path with latency per hop - essential for debugging distributed latency.
- Define SLOs and alert on themAlerting on every spike creates alert fatigue. SLOs (Service Level Objectives) define what 'good enough' is - alert only when it's actually broken from the user's perspective.
Output Format
Full observability stack: structured logs → Prometheus metrics → Grafana dashboards → OpenTelemetry traces → SLO-based alerts.
Three Pillars:
Logs: JSON, pino/structlog → Loki/CloudWatch
Metrics: /metrics endpoint → Prometheus → Grafana
Traces: OpenTelemetry → Jaeger/Tempo
Alerts:
Error rate > 1% → PagerDuty (critical)
P99 > 500ms → Slack (warning)Examples
Input
Output
Edge Cases & Gotchas
- Cardinality explosion in Prometheus - Labels with high-cardinality values (user ID, request ID) create millions of unique time series and crash Prometheus. Never use user IDs as label values.
- Log sampling - High-traffic services generate billions of log lines. Sample at 1-10% for debug logs; always log errors at 100%. Use head-based or tail-based sampling for traces.
- Clock skew in distributed traces - Nodes with out-of-sync clocks produce overlapping or impossible trace spans. Use NTP or PTP and handle clock skew in trace analysis tools.
- Alert fatigue - Too many low-severity alerts train engineers to ignore them all. Keep critical alerts actionable and urgent; route informational alerts to a dashboard only.
When to Break These Rules
- Use Datadog or New Relic instead of self-hosted Prometheus/Grafana when the team doesn't want to manage the observability infrastructure - they cost more but save significant ops time.
- Use AWS CloudWatch when already on AWS and don't want to manage a separate observability stack - it's less capable but zero operational overhead.
Reference Files
https://opentelemetry.io/docs/https://prometheus.io/docs/