Module 4 · Secrets Management at Scale 🔒

Secrets management is the discipline of generating, storing, distributing, rotating, and auditing access to credentials — API keys, database passwords, encryption keys, OAuth tokens. At scale (hundreds of services, dozens of secrets each, rotation requirements), getting this right requires infrastructure, not Git-committed YAML. This module covers the patterns, tools, and operational practices.

Why “just put them in env vars” doesn’t scale

• Where do they come from? Manually entered in CI? Stored in encrypted git? Both have problems
• Rotation is manual — secrets stay the same for years; one leak compromises permanently
• No audit — who read what secret when? Unknowable
• No revocation — leaked secret stays valid until manually rotated
• Dev/staging/prod sprawl — same secret reused across environments

Secrets management evolution

1. Stage 0: hardcoded in code or committed to git. Don’t
2. Stage 1: environment variables set on deploy. Better; secrets out of code. But where do env vars come from?
3. Stage 2: centralized secret store; CI/runtime fetches from store. AWS Secrets Manager, HashiCorp Vault, etc.
4. Stage 3: dynamic secrets — generated on-demand, short TTL, no long-lived credentials at all
5. Stage 4: identity-based access — workloads authenticate via cloud-native identity (IAM roles, SPIFFE, workload identity), no fetched secrets

Most teams sit between stages 2 and 3. Identity-based (stage 4) is the modern target.

Tools — the major players

Cloud-native

• AWS Secrets Manager — auto-rotation built-in for RDS; integrates with KMS
• AWS Systems Manager Parameter Store — cheaper, simpler; less rotation automation
• Azure Key Vault — secrets, keys, certs in one service
• GCP Secret Manager — straightforward; integrates with IAM

Self-hosted

• HashiCorp Vault — most flexible; PKI, dynamic secrets, transit encryption, multiple backends
• Bitwarden / Vaultwarden — primarily for human secrets; can do machine secrets
• Conjur (CyberArk) — enterprise-focused

Kubernetes-specific

• Kubernetes Secrets — built-in but base64’d not encrypted by default; needs etcd encryption + careful RBAC
• Sealed Secrets — encrypt secrets, commit to git, only the cluster can decrypt
• External Secrets Operator — sync from external store (Vault, AWS, etc.) into K8s Secrets
• Secrets Store CSI Driver — mount secrets directly from external store as files

Code-level

• SOPS (Mozilla) — encrypt files in git with KMS / age / GPG; decrypt in CI
• git-crypt — transparent file encryption in git
• age — modern simple file encryption tool; pairs well with SOPS

Vault — the reference implementation

HashiCorp Vault is worth understanding because its concepts (auth methods, secret engines, policies, leases, dynamic secrets) appear everywhere.

# Configure database secret engine
vault secrets enable database
vault write database/config/postgres-prod \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@db.example.com:5432/" \
  allowed_roles="readonly,readwrite" \
  username="vault_admin" \
  password="..."

# Define a role that creates dynamic credentials
vault write database/roles/readonly \
  db_name=postgres-prod \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
                       GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# App requests a credential — gets a unique short-lived user
vault read database/creds/readonly
# Returns: {"username": "v-readonly-x9f2", "password": "..."}
# Auto-revoked after 1h

Dynamic secrets are the model: app gets a credential created just for this session, valid for an hour, then revoked. Compromise window is bounded.