Designing a Webhook Delivery System for 10 Million Events a Day

The Problem & Industry Shift

Webhooks have become the de facto standard for real-time event notifications in modern APIs. From payment gateways to CI/CD pipelines, services rely on webhooks to push events to consumers. However, as your platform scales to 10 million events per day (roughly 115 events per second on average, with peaks much higher), the naive approach of synchronous HTTP calls from your main application thread breaks down.

Why traditional webhook delivery fails at scale:

Tight coupling: Direct HTTP calls in the request lifecycle block your API and couple availability to third-party endpoints.
No retry logic: Transient network failures or consumer downtime cause data loss.
No ordering or deduplication: Duplicate events or out-of-order delivery corrupt consumer state.
No observability: You can't answer "which events failed and why?"

Industry leaders like Stripe, GitHub, and Shopify have moved to queue-based, asynchronous delivery with robust retry and idempotency mechanisms. This article details how to design such a system, drawing on patterns from Stripe's webhook documentation and GitHub's webhook events.

Architecture & Core Mechanics

A scalable webhook delivery system decouples event production from delivery using a message queue. Here's the high-level flow:

[Your Service] -> [Event Bus] -> [Dispatcher] -> [Retry Queue] -> [Consumer]
                     |                |
                     |                +---> [DLQ] (dead letter queue)
                     +---> [Event Store]

Components:

Event Producer: Your application emits domain events (e.g., order.created, payment.failed).
Event Bus: A durable message queue (e.g., AWS SQS, RabbitMQ, Apache Kafka) that buffers events. This ensures no data loss if downstream fails.
Dispatcher: A worker process that reads events from the queue and performs the HTTP POST to the consumer's webhook URL.
Retry Queue: Events that fail are moved to a retry queue with exponential backoff.
Dead Letter Queue (DLQ): After maximum retries, events are parked for manual inspection.
Event Store: A database (e.g., PostgreSQL) that stores the event payload and delivery status for audit and replay.

Key design decisions:

At-least-once delivery: Queues guarantee at-least-once, so consumers must be idempotent.
Ordering: If order matters, use a partitioned queue (e.g., Kafka partitions by event ID or consumer ID).
Payload signing: Use HMAC signatures to authenticate webhook payloads (see GitHub's guide).
Backpressure: Use a bounded queue and concurrency limits to avoid overwhelming consumers.

Production Code Example

Below is a TypeScript implementation of a dispatcher using AWS SQS and Node.js. It demonstrates critical engineering decisions: idempotency, retry with exponential backoff, and signature generation.

// dispatcher.ts
import { SQSClient, SendMessageCommand, DeleteMessageCommand } from "@aws-sdk/client-sqs";
import { createHmac } from "crypto";

const sqs = new SQSClient({ region: process.env.AWS_REGION });
const QUEUE_URL = process.env.WEBHOOK_QUEUE_URL!;
const DLQ_URL = process.env.WEBHOOK_DLQ_URL!;
const SECRET = process.env.WEBHOOK_SECRET!;

interface WebhookEvent {
  id: string; // unique event ID for idempotency
  type: string;
  payload: unknown;
  targetUrl: string;
  attempt: number;
}

// Send event to queue
async function enqueueEvent(event: WebhookEvent) {
  const params = {
    QueueUrl: QUEUE_URL,
    MessageBody: JSON.stringify(event),
    MessageDeduplicationId: event.id, // for FIFO queues
    MessageGroupId: event.id, // maintain order per event
  };
  await sqs.send(new SendMessageCommand(params));
}

// Dispatcher worker
async function processMessage(message: any) {
  const event: WebhookEvent = JSON.parse(message.Body);

  // Idempotency check: store processed event IDs in Redis or DB
  if (await isProcessed(event.id)) {
    console.log(`Event ${event.id} already processed, skipping.`);
    await deleteMessage(message);
    return;
  }

  // Generate HMAC signature
  const signature = createHmac("sha256", SECRET)
    .update(JSON.stringify(event.payload))
    .digest("hex");

  try {
    const response = await fetch(event.targetUrl, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "X-Webhook-Signature": `sha256=${signature}`,
        "X-Event-ID": event.id,
      },
      body: JSON.stringify(event.payload),
      signal: AbortSignal.timeout(5000), // 5s timeout
    });

    if (!response.ok) {
      throw new Error(`HTTP ${response.status}`);
    }

    // Mark as processed
    await markProcessed(event.id);
    await deleteMessage(message);
  } catch (error) {
    console.error(`Delivery failed for event ${event.id}:`, error);
    await handleRetry(event, message);
  }
}

// Exponential backoff retry
async function handleRetry(event: WebhookEvent, message: any) {
  const maxRetries = 5;
  if (event.attempt >= maxRetries) {
    // Move to DLQ
    await sqs.send(new SendMessageCommand({
      QueueUrl: DLQ_URL,
      MessageBody: JSON.stringify(event),
    }));
    await deleteMessage(message);
    console.error(`Event ${event.id} moved to DLQ after ${maxRetries} attempts.`);
    return;
  }

  // Increase attempt count and send to retry queue with delay
  const retryEvent = { ...event, attempt: event.attempt + 1 };
  const delay = Math.min(300, Math.pow(2, retryEvent.attempt) * 10); // 10s, 20s, 40s...
  await sqs.send(new SendMessageCommand({
    QueueUrl: QUEUE_URL,
    MessageBody: JSON.stringify(retryEvent),
    DelaySeconds: delay,
  }));
  await deleteMessage(message);
}

// Helper functions (simplified)
async function isProcessed(id: string): Promise<boolean> { /* check Redis */ return false; }
async function markProcessed(id: string): Promise<void> { /* set Redis */ }
async function deleteMessage(message: any) {
  await sqs.send(new DeleteMessageCommand({
    QueueUrl: QUEUE_URL,
    ReceiptHandle: message.ReceiptHandle,
  }));
}

Key points:

Idempotency: Use event ID to skip duplicates. This is crucial because queues deliver at-least-once.
Timeout: AbortSignal.timeout prevents hanging requests.
Retry with backoff: Exponential backoff with jitter (not shown) prevents thundering herd.
DLQ: After max retries, events are isolated for debugging.

Performance, Cost & Trade-offs

Latency vs. Throughput:

Queue-based delivery adds latency (typically 100-500ms) compared to synchronous calls, but it decouples your system and improves reliability.
To meet 10M events/day, you need to process ~115 events/sec average. With a dispatcher concurrency of 20 workers, each taking 200ms, you can handle 100 req/sec. Scale horizontally with more workers.

Cost considerations:

SQS costs: $0.40 per million requests after free tier. For 10M events, that's ~$4/month for queue operations.
Compute: Running dispatcher workers on EC2 or Lambda. Lambda scales automatically but may have timeout limits.
Database: Storing event logs for audit can grow large; use retention policies.

Trade-offs:

At-least-once vs. exactly-once: True exactly-once is impossible in distributed systems. At-least-once + idempotency is the pragmatic standard.
Ordering: FIFO queues guarantee order but limit throughput (300 msg/s). For 10M/day, you may need multiple FIFO queues or accept best-effort ordering.
Security: Signing payloads adds CPU overhead but is essential. Use a constant-time comparison to avoid timing attacks.

Benchmarks: In a typical setup with SQS and Node.js, you can achieve 500-1000 events/sec per dispatcher instance with moderate CPU usage. Monitor memory and network to tune concurrency.

Actionable Checklist / Summary

When adopting a webhook delivery system for high throughput, follow these steps:

Use a durable queue (SQS, RabbitMQ, Kafka) to buffer events and decouple producers from consumers.
Implement idempotency at the consumer side using a unique event ID. Store processed IDs in Redis or a database.
Design retry with exponential backoff and jitter. Set a maximum retry count (e.g., 5) and move failed events to a DLQ.
Sign payloads with HMAC and provide a signature header. Validate signatures in your consumer to prevent forgery.
Set timeouts on HTTP calls (e.g., 5 seconds) to avoid hanging workers.
Monitor everything: Track delivery success rates, retry counts, DLQ depth, and consumer latency. Use structured logging.
Scale horizontally: Run multiple dispatcher instances behind a queue. Use auto-scaling based on queue depth.
Document your webhook contract for consumers, including retry behavior and event schemas.

The Problem & Industry Shift

Architecture & Core Mechanics

Production Code Example

Performance, Cost & Trade-offs

Actionable Checklist / Summary

References