DevOps5 min read

Secrets Management

Manage secrets securely across environments - HashiCorp Vault, AWS Secrets Manager, External Secrets Operator, SOPS, and secret rotation.

Secrets ManagementHashiCorp VaultAWS Secrets ManagerSOPSSealed Secrets

skill: Secrets Management category: DevOps tags: [Secrets Management, HashiCorp Vault, AWS Secrets Manager, SOPS, Sealed Secrets] readingTime: 5

Secrets (API keys, database passwords, TLS certificates) must never appear in code, git history, environment variable dumps, or logs. The goal is short-lived, auto-rotated credentials accessed programmatically - not long-lived secrets passed around as strings. Zero-trust means applications get secrets based on identity, not shared passwords.

Workflow

Store secrets in AWS Secrets ManagerAWS Secrets Manager stores secrets encrypted at rest, controls access with IAM, provides rotation, and audits every access via CloudTrail.
Use HashiCorp Vault for dynamic secretsVault generates short-lived credentials dynamically - your database never has a permanent password. The credential expires after the TTL, limiting breach damage.
Inject secrets into Kubernetes with External Secrets OperatorExternal Secrets Operator bridges Kubernetes Secrets with external stores (AWS Secrets Manager, Vault, GCP Secret Manager) - pods get Kubernetes Secrets without storing secrets in git.
Encrypt secrets in git with SOPSSOPS (Secrets OPerationS) encrypts secret files with AWS KMS or age keys - the encrypted file lives in git, only decryptable by authorized IAM roles or key holders.
Audit and rotate secrets regularlyStale credentials are the most common source of breaches - an API key rotated never is a credential waiting to be stolen. Automate rotation and audit access.

Output Format

Secrets: AWS Secrets Manager for storage + rotation, Vault for dynamic DB credentials, External Secrets Operator for Kubernetes injection, SOPS for git encryption, gitleaks for leak detection.

AWS SM: create-secret + automatic rotation + IAM role access
Vault: database/creds/role (dynamic, expires in 1h)
External Secrets: SecretStore + ExternalSecret → Kubernetes Secret
SOPS: .sops.yaml config + sops --encrypt + sops --decrypt | kubectl apply
CI: gitleaks pre-commit + trufflehog on PR

Examples

Input

Output

Edge Cases & Gotchas

Secret in git history - Once committed, a secret in git history exists forever. Use BFG Repo Cleaner or git-filter-repo to purge, then force-push and notify all contributors to re-clone. Rotate the secret immediately.
Environment variable leakage - Many debugging tools (crash reporters, APM agents) capture environment variables. Secrets in env vars get sent to third-party services. Prefer in-memory secrets fetched at runtime from Vault/Secrets Manager.
Vault seal/unseal - Vault "seals" itself if the process restarts - it won't serve secrets until manually unsealed. Use auto-unseal with AWS KMS or Azure Key Vault to avoid operational downtime.
Cross-account secret sharing - AWS Secrets Manager resource policies allow cross-account access. Don't duplicate secrets across accounts - share them with IAM resource policies from the owning account.

When to Break These Rules

Use environment variables for local development (dotenv) - Vault/Secrets Manager overhead is too high for a solo developer's local machine.
Use Kubernetes Secrets directly (base64-encoded) for small clusters where the overhead of External Secrets Operator isn't justified - enable envelope encryption on the etcd cluster for security.

Reference Files

https://developer.hashicorp.com/vault/docs
https://external-secrets.io/