Linux Administration
Administer Linux servers - systemd services, process management, disk I/O, network configuration, security hardening, and performance tuning.
skill: Linux Administration category: DevOps tags: [Linux, System Administration, Shell, systemd, Performance Tuning] readingTime: 6
Linux system administration is about understanding what the OS is doing and why. Symptoms (high CPU, full disk, slow I/O) have specific diagnostic commands. Read system logs (journalctl), resource usage (top, htop, iostat, vmstat), and network state (ss, netstat, tcpdump) before changing anything.
Workflow
- Manage services with systemdsystemd is the init system on all modern Linux distributions - it manages service lifecycle, logging (journald), and boot order.
- Investigate high CPU and memoryWhen a server is slow, the first question is what's consuming resources. Know the tools for CPU, memory, and I/O investigation.
- Configure networking and firewallNetwork misconfiguration is a common security vulnerability and cause of connectivity issues. Understand interface state, routing, and firewall rules.
- Harden SSH accessSSH is the primary attack vector for Linux servers. Disable password auth, use key-based auth, and restrict root login.
- Tune kernel parameters for productionDefault kernel parameters are conservative. Production servers handling many connections need tuned network buffers, file descriptor limits, and TCP settings.
Output Format
Linux admin: systemd service unit files, top/iostat/ss for investigation, nftables firewall, SSH hardening with key-only auth, and sysctl tuning for production.
systemctl enable --now myapp.service
journalctl -u myapp -f
top → pidstat → iostat → ss -tlnp
nft -f /etc/nftables.conf
sysctl -p /etc/sysctl.d/99-production.confExamples
Input
Output
Edge Cases & Gotchas
- OOM killer terminating your process - Linux's OOM killer kills the process with highest OOM score when RAM is full. To protect critical services: echo -17 > /proc/$(pidof myapp)/oom_adj or set OOMScoreAdjust=-1000 in the systemd unit.
- Inode exhaustion - df -i shows inode usage. You can run out of inodes (file slots) while still having disk space - common with many small files (log directories, email spools). rm files or use tune2fs to resize.
- File descriptor limits - Web servers and databases need high FD limits. Limits set in /etc/security/limits.conf only apply to PAM sessions (SSH). systemd services need LimitNOFILE in the unit file.
- CPU steal in cloud VMs - %st in top shows CPU stolen by the hypervisor from other tenants. High steal (>5%) means the host is overcommitted - move to a larger instance or a dedicated host.
When to Break These Rules
- Use managed services (RDS, Elasticache, Cloud Run) instead of self-administering Linux servers when possible - every minute spent on OS administration is a minute not spent on product.
- Use Ansible or Chef instead of manual sshd_config edits for fleet-wide configuration management - manual changes don't scale and drift from desired state.
Reference Files
https://www.man7.org/linux/man-pages/https://www.brendangregg.com/linuxperf.html