Zero-Downtime Kubernetes Deployments: A Production Runbook

Why Zero-Downtime Matters More Than You Think

Every second of downtime during a deployment is a second your users experience degraded service. For our aviation client processing 1,000+ daily flights, even a 30-second deployment gap could mean missed schedule updates affecting thousands of passengers.

Rolling Update Pitfalls

Kubernetes rolling updates seem safe but hide subtle failure modes. The default maxUnavailable: 25% means during a bad deploy, 25% of your pods serve broken responses before health checks catch the problem. With a 30-second health check interval, that's 30 seconds of partial failure.

apiVersion: apps/v1
kind: Deployment
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0  # Never remove old pods before new ones are ready
  template:
    spec:
      containers:
        - name: app
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 3
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 15"]  # Drain connections

Canary with Automated Analysis

Our production pattern uses Argo Rollouts with automated canary analysis. New versions receive 5% of traffic initially, with Prometheus metrics determining whether to promote or rollback automatically.

The analysis template checks three signals: error rate must stay below 0.1%, p99 latency must not increase more than 15%, and the pod restart count must remain at zero. If any metric breaches its threshold during the 10-minute observation window, the rollout automatically reverts without human intervention.

Connection Draining

The most overlooked cause of deployment errors is in-flight request termination. When Kubernetes sends SIGTERM to a pod, active HTTP connections are severed immediately unless you implement graceful shutdown. Our standard pattern: the preStop hook sleeps for 15 seconds while the application stops accepting new connections and drains existing ones. This ensures every in-flight request completes before the pod terminates.

Production Checklist

Before every production deployment, verify: Pod Disruption Budgets prevent simultaneous pod termination, readiness probes accurately reflect application state, graceful shutdown handlers drain all connection types (HTTP, WebSocket, gRPC), and rollback procedures are tested within the last 30 days.