DevOps6 min read

Log Management

Collect, store, search, and alert on application logs at scale - using structured logging, log aggregation (ELK/Loki), and retention policies.

Log ManagementELK StackLokiStructured LoggingObservability

skill: Log Management category: DevOps tags: [Log Management, ELK Stack, Loki, Structured Logging, Observability] readingTime: 6

Logs are the first thing you reach for when something breaks. Without structured, centralized, searchable logs, debugging production issues takes hours instead of minutes. Structure from day one - retrofitting JSON logging onto a million-line plaintext log archive is painful.

Workflow

Write structured JSON logsJSON logs are machine-parseable - every field is queryable without regex. Structured logs enable log-based metrics, alerting, and correlation across services.
Collect with Promtail or FluentdApplications write to stdout/files; a log collector ships those logs to your aggregation system. Decoupling collection from the app avoids blocking on I/O.
Search and query with Kibana or GrafanaA log aggregator without a search UI is a black box. Kibana (ELK) and Grafana (Loki) provide full-text search, field filtering, and dashboards.
Set up log-based alertsLogs surface business events (payment failures, auth errors) that metrics don't capture. Alert on error spikes or critical business events.
Define retention and compliance policiesLogs grow at 10-100GB/day for busy services. Indefinite retention is cost-prohibitive; many regulations require specific minimum retention periods.

Output Format

Centralized logging pipeline: structured JSON → Promtail/Fluentd → Loki/Elasticsearch → Grafana/Kibana dashboards + alerts.

Log Pipeline:
  Emit:    JSON via pino/structlog
  Collect: Promtail reads stdout/log files
  Store:   Loki (label-indexed) or Elasticsearch (full-text)
  Query:   LogQL (Loki) or KQL/Lucene (Kibana)
  Alert:   Error rate > threshold → PagerDuty
  Retain:  30 days hot, 1 year cold (S3 Glacier)

Examples

Input

Output

Edge Cases & Gotchas

Logging sensitive data - Logs are often less protected than databases. Never log passwords, full credit card numbers, SSNs, or API keys. Redact with patterns before shipping.
Log volume in debug mode - Debug logging in production generates 100x more volume and can degrade app performance. Use LOG_LEVEL=info in production; debug only per-request via feature flag.
Fluentd buffer overflow - If the log destination is slow or down, Fluentd's buffer fills and logs are dropped. Set buffer chunk_limit_size and queue_limit_length, and use a persistent disk buffer.
Timezone in log timestamps - Always log timestamps in UTC. Mixed timezones in logs across services make correlation analysis impossible.

When to Break These Rules

Use a managed logging service (Datadog Logs, Splunk, AWS CloudWatch) instead of self-hosted ELK when the ops overhead of managing an Elasticsearch cluster is too high.
Use sampling for very high-throughput debug logs (> 1M events/sec) - logging every event at that volume is cost-prohibitive even in Loki.

Reference Files

https://grafana.com/docs/loki/latest/
https://www.elastic.co/guide/en/elasticsearch/reference/current/