Performance Monitoring & APM
Implement APM: Sentry for error tracking, OpenTelemetry traces, custom metrics with Prometheus, SLI/SLO definitions, and performance alerting.
Application Performance Monitoring (APM) provides visibility into how your system performs in production. It combines error tracking, distributed tracing, metrics, and alerting. Without APM, production issues are discovered by users, not engineers.
Instrument before you need it - adding monitoring during an outage is too late. Start with Sentry (errors), then add OpenTelemetry traces, then custom business metrics. Define SLOs for services you own. Alert on symptoms (high error rate), not causes (CPU usage).
Workflow
- Set up Sentry for error trackingSentry captures unhandled errors with full stack traces, context, and user impact data.
- Instrument with OpenTelemetryDistributed traces show the entire request path across services - essential for microservices debugging.
- Define SLIs and SLOsSLOs translate business reliability requirements into measurable targets and alert thresholds.
- Instrument custom Prometheus metricsBusiness metrics (order rate, payment success, queue depth) provide context that infrastructure metrics can't.
- Set up alerting and runbooksAlerts without context create alert fatigue; runbooks tell on-call what to do.
Output Format
Sentry with source maps and user context, OpenTelemetry auto-instrumentation + collector, SLO definitions with error budget alerts, Prometheus custom metrics, AlertManager with Slack/PagerDuty routing.
Examples
Custom span: `const tracer = trace.getTracer("checkout"); const span = tracer.startActiveSpan("process-payment", async (span) => { try { span.setAttributes({ "payment.method": "stripe", "order.amount": order.total }); const result = await stripe.charge(order); span.setStatus({ code: SpanStatusCode.OK }); return result } catch (e) { span.setStatus({ code: SpanStatusCode.ERROR, message: e.message }); throw e } finally { span.end() } })`SLO burn rate: `alert: ErrorBudgetBurnRate14x expr: sum(rate(http_request_duration_seconds_count{status_code=~"5.."}[1h])) / sum(rate(http_request_duration_seconds_count[1h])) > 14 * 0.001 // 14x burn of 0.1% error budget for: 2m labels: severity: critical`Prometheus: `const queueDepth = new Gauge({ name: "job_queue_depth", help: "Current queue depth", labelNames: ["queue_name"] }); setInterval(async () => { const depth = await queue.getWaitingCount(); queueDepth.set({ queue_name: "email" }, depth) }, 15000)`Edge Cases & Gotchas
- High cardinality labels: Prometheus label values create a time series per unique combination. Never use `userId` or `requestId` as labels - use aggregation-friendly values (endpoint, status code, region)
- Sampling traces: tracing every request at high QPS is expensive. Use head-based sampling (sample X% at ingestion) or tail-based sampling (sample 100% of errors, X% of successes)
- Alert fatigue: >5 active alerts/week that resolve without human action means your thresholds are too sensitive. Regularly review and tune alert rules
When to Break These Rules
- For early-stage apps with <100 users, CloudWatch or a simple error email is sufficient. Full APM stack setup is justified when production incidents start taking >1h to diagnose.
Reference Files
Sentry docs: https://docs.sentry.io/OpenTelemetry Node: https://opentelemetry.io/docs/instrumentation/js/Prometheus best practices: https://prometheus.io/docs/practices/naming/SLO workbook: https://sre.google/workbook/alerting-on-slos/