Implementing Graceful Shutdown and Connection Draining in Go

When deploying updates to production services running behind a load balancer or inside Kubernetes, rolling deployments are intended to be completely invisible to users. Yet many engineering teams still observe intermittent 502 Bad Gateway errors during redeployments.

In almost all cases, the root cause is premature process termination: the application receives a termination signal (SIGTERM) and exits immediately, aborting active HTTP requests and severing open TCP connections.

Achieving true zero-downtime deployments requires implementing graceful shutdown and understanding the lifecycle of a termination signal.

The Shutdown Sequence

When a container orchestrator replaces an existing pod with a new replica, the following steps occur concurrently:

1. Endpoint Removal: The container orchestrator removes the pod's IP address from the service endpoints and ingress load balancer.
2. Termination Signal: A SIGTERM is sent to the process inside the container.
3. Grace Period Clock: The orchestrator starts a countdown (typically 30 seconds). If the process is still running when the timer expires, it receives an uncatchable SIGKILL.

The critical catch is that endpoint removal across distributed load balancer nodes is not instantaneous—it can take several seconds for routing tables to update. If your application terminates the instant it receives SIGTERM, new requests that were already in flight from the load balancer will hit a closed socket.

Production Go Implementation

Here is an idiomatic pattern in Go using signal.NotifyContext and http.Server.Shutdown:

package main

import (
	"context"
	"errors"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
		w.Write([]byte("ok"))
	})
	mux.HandleFunc("/api/work", func(w http.ResponseWriter, r *http.Request) {
		// Simulate in-flight workload
		time.Sleep(2 * time.Second)
		w.Write([]byte("processed"))
	})

	srv := &http.Server{
		Addr:         ":8080",
		Handler:      mux,
		ReadTimeout:  10 * time.Second,
		WriteTimeout: 10 * time.Second,
		IdleTimeout:  60 * time.Second,
	}

	// Listen for termination signals from the OS or orchestrator
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()

	go func() {
		log.Printf("Server listening on %s", srv.Addr)
		if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
			log.Fatalf("HTTP server failed: %v", err)
		}
	}()

	// Block until a signal is received
	<-ctx.Done()
	log.Println("Shutdown signal received, initiating graceful drain...")

	// Create a deadline context for the server to finish in-flight requests
	shutdownCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
	defer cancel()

	// Stop accepting new connections and wait for active requests to complete
	if err := srv.Shutdown(shutdownCtx); err != nil {
		log.Printf("Graceful shutdown failed, forcing close: %v", err)
		_ = srv.Close()
	}

	log.Println("Server cleanly stopped")
}

Key Engineering Nuances

Notice three important details in this implementation:

• http.ErrServerClosed Check: When srv.Shutdown is invoked, srv.ListenAndServe immediately returns http.ErrServerClosed. This is an expected lifecycle event and should not be logged as a fatal failure.
• Dedicated Timeout Context: The shutdown context has an explicit deadline (15 seconds in this example). This ensures that a stuck database query or hung third-party connection cannot prevent the process from terminating before the container orchestrator sends a hard SIGKILL.
• Closing Downstream Resources: Once srv.Shutdown completes, you should cleanly close database connection pools, Redis clients, and flush buffered loggers or telemetry spans before the application process exits.