Gyaan

Secrets Management and TLS

advanced secrets vault tls certificates encryption

Here’s a nightmare scenario: someone pushes an .env file with the production database password to a public GitHub repo. A bot scrapes it within minutes. The database is compromised. This happens more often than we’d like to admit.

Secrets management is about making sure passwords, API keys, tokens, and certificates never end up in code, Git history, or plain-text config files.

The Problem with Environment Variables

The most common approach is environment variables. They’re simple, but they have real limitations:

  • Env vars are visible to any process on the machine (/proc/<pid>/environ on Linux).
  • They get logged accidentally — a debug log that prints all env vars leaks every secret.
  • No access control — if we can SSH into the machine, we can see all of them.
  • No audit trail — who accessed which secret? No idea.
  • No rotation — changing a secret means redeploying every service that uses it.

Env vars are fine for development. For production, we need something better.

Approaches to Secrets Management

1. Mounted files / volumes — secrets are written to files that containers mount. Better than env vars because we can restrict file permissions, but still static.

2. Cloud-native secrets managers — AWS Secrets Manager, GCP Secret Manager, Azure Key Vault. They store secrets encrypted, provide access control, audit logs, and automatic rotation.

3. HashiCorp Vault — the most popular open-source option. Cloud-agnostic, works everywhere.

HashiCorp Vault Basics

Vault is a tool for securely storing and accessing secrets. It encrypts everything at rest, provides fine-grained access policies, keeps an audit log of every access, and can even generate short-lived credentials on the fly.

# Store a secret
vault kv put secret/myapp/db \
  username="admin" \
  password="s3cur3-p@ss"

# Read it back
vault kv get secret/myapp/db

# Read just one field
vault kv get -field=password secret/myapp/db

Vault also supports dynamic secrets — instead of storing a static database password, Vault creates a new temporary database user on demand. When the lease expires, Vault deletes the user. If credentials leak, they’re already expired. This is a game changer for security.

Kubernetes Secrets

Kubernetes has a built-in Secret resource. It’s better than hardcoding, but has a catch — Secrets are only base64-encoded, not encrypted (by default):

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  username: YWRtaW4=      # base64 of "admin"
  password: czNjdXIzLXBAc3M=  # base64 of "s3cur3-p@ss"

Anyone with cluster access can decode these. For real security, we pair K8s Secrets with Sealed Secrets (encrypts secrets in Git, only the cluster can decrypt) or integrate with Vault using the Vault Agent Injector.

TLS Refresher

TLS (Transport Layer Security) encrypts data in transit. Every HTTPS connection uses TLS. Here’s the quick breakdown:

  • Certificate — a file that proves “I am who I say I am.” Contains a public key, signed by a trusted authority.
  • Certificate Authority (CA) — the trusted party that signs certificates. Browsers trust a handful of CAs (Let’s Encrypt, DigiCert, etc.).
  • Private key — stays on our server, never shared. Used to decrypt data encrypted with our public key.
  • The handshake — client connects, server presents certificate, client verifies it’s signed by a trusted CA, they agree on encryption keys, data flows encrypted.

cert-manager for Auto-Renewal

Managing TLS certificates manually is painful and error-prone. cert-manager automates this in Kubernetes. It requests certificates from Let’s Encrypt (or other CAs), installs them, and renews them before they expire — all automatically.

# cert-manager issuer for Let's Encrypt
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
      - http01:
          ingress:
            class: nginx

Outside Kubernetes, tools like Caddy (which this very site uses!) handle TLS certificates automatically too.

mTLS — Mutual TLS

Normal TLS is one-way: the client verifies the server. mTLS (mutual TLS) goes both ways — the server also verifies the client. Both sides present certificates.

This is huge for service-to-service communication. Instead of relying on network firewalls alone, each service proves its identity cryptographically. Service meshes like Istio and Linkerd set up mTLS automatically between all services in a Kubernetes cluster.

Secrets Rotation

Even with a secrets manager, secrets should be rotated regularly. If a credential leaks, the blast radius depends on how old it is:

  • Automated rotation — Vault and cloud secrets managers can rotate passwords on a schedule.
  • Short-lived credentials — better than rotation. If a token expires in 1 hour, a leak is only dangerous for 1 hour.
  • Rotation without downtime — support multiple active versions of a secret during rotation so services can gracefully switch over.

In simple language, secrets management boils down to three rules: never put secrets in code, always encrypt them at rest and in transit, and rotate them regularly. Tools like Vault make this practical, and TLS/mTLS makes sure data is encrypted as it moves between services.

References