Designing a Webhook Delivery System for 10 Million Events a Day

The Problem & Industry Shift

Webhooks have become the backbone of modern event-driven architectures, enabling real-time integrations between services. From payment notifications to CI/CD triggers, systems rely on webhooks to deliver critical events. However, as your platform scales, naive webhook delivery—simple HTTP POSTs from your application—breaks down under load. At LiveReview, we faced the challenge of delivering AI code review events to thousands of repositories, with peaks of 10 million events per day. The naive approach led to timeouts, lost events, and blocked workers.

The industry shift is toward event-driven architectures where webhooks are treated as first-class citizens, with dedicated delivery pipelines that guarantee at-least-once delivery, handle retries, and provide observability. This article details the architecture and code patterns you need to build such a system.

Architecture & Core Mechanics

A robust webhook delivery system decouples event production from delivery using a queue. The core components are:

1. Producer: Generates events (e.g., code review completed) and enqueues them.
2. Queue: Buffers events, providing durability and backpressure. (e.g., SQS, RabbitMQ, Kafka)
3. Dispatcher: Pulls events from the queue and sends HTTP requests to the configured endpoint.
4. Retry Engine: Handles failures with exponential backoff and dead-letter queues.
5. Observability: Metrics, logging, and tracing to monitor delivery health.

Below is a high-level flow:

[Producer] --> (Event) --> [Queue] --> [Dispatcher] --> HTTP POST --> [Consumer]
                                 |         |
                                 |         +---> Failure? --> [Retry Engine] --> Backoff --> Dispatcher
                                 |                                    |
                                 +---> Dead Letter Queue (after max retries)

Key design decisions:

• Queue choice: For high throughput (10M/day ≈ 115 msg/sec average, but peaks higher), a managed queue like AWS SQS or Google Pub/Sub is preferable for durability and scalability. Kafka offers replayability but adds operational complexity.
• Batching: To reduce API calls, batch events per consumer endpoint when possible, but respect consumer limits.
• Idempotency: Consumers may receive duplicates due to retries. Include an event_id in the payload and require consumers to deduplicate.
• Security: Sign payloads with HMAC and validate on the consumer side to prevent tampering.

Production Code Example

Below is a TypeScript implementation of a dispatcher with retry logic, using a generic queue interface. This code runs as a worker process.

// types.ts
export interface WebhookEvent {
  id: string; // unique event ID
  type: string;
  payload: unknown;
  endpoint: string; // consumer URL
  secret: string; // for HMAC signing
}

// queue.ts - abstraction over SQS, RabbitMQ, etc.
export interface Queue {
  receive(): Promise<WebhookEvent[]>;
  delete(receiptHandle: string): Promise<void>;
  sendToDeadLetter(event: WebhookEvent, reason: string): Promise<void>;
}

// dispatcher.ts
import crypto from 'crypto';
import { Queue, WebhookEvent } from './types';

const MAX_RETRIES = 5;
const BASE_DELAY_MS = 1000; // 1 second

async function sendWebhook(event: WebhookEvent): Promise<boolean> {
  const body = JSON.stringify(event);
  const signature = crypto
    .createHmac('sha256', event.secret)
    .update(body)
    .digest('hex');

  const controller = new AbortController();
  const timeout = setTimeout(() => controller.abort(), 5000); // 5s timeout

  try {
    const response = await fetch(event.endpoint, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'X-Webhook-Signature': signature,
        'X-Event-ID': event.id,
      },
      body,
      signal: controller.signal,
    });
    return response.ok; // 2xx status
  } catch (error) {
    console.error(`Failed to send event ${event.id}:`, error);
    return false;
  } finally {
    clearTimeout(timeout);
  }
}

async function processEvent(event: WebhookEvent, attempt: number): Promise<void> {
  const success = await sendWebhook(event);
  if (success) {
    console.log(`Event ${event.id} delivered`);
    return;
  }

  if (attempt >= MAX_RETRIES) {
    console.error(`Event ${event.id} failed after ${MAX_RETRIES} attempts`);
    await queue.sendToDeadLetter(event, 'Max retries exceeded');
    return;
  }

  // Exponential backoff with jitter
  const delay = BASE_DELAY_MS * Math.pow(2, attempt) + Math.random() * 1000;
  console.log(`Retrying event ${event.id} in ${delay}ms (attempt ${attempt + 1})`);
  setTimeout(() => processEvent(event, attempt + 1), delay);
}

export async function startDispatcher(queue: Queue): Promise<void> {
  while (true) {
    const events = await queue.receive();
    await Promise.all(events.map(event => processEvent(event, 0)));
    // In production, delete from queue only after successful processing (or after dead-lettering)
    // Here we assume processEvent handles deletion internally.
  }
}

Critical engineering decisions:

• Timeout: Use a 5-second timeout to avoid hanging connections.
• Retry with jitter: Prevents thundering herd on retries.
• Idempotency: The X-Event-ID header allows consumers to deduplicate.
• Dead-letter queue: After max retries, events are parked for manual inspection.

Performance, Cost & Trade-offs

Latency vs. Throughput: With a queue, you introduce a small latency (milliseconds) but gain massive throughput. For 10M events/day, a single dispatcher can handle ~100 events/sec, but you'll need multiple workers. Horizontal scaling is straightforward by increasing the number of dispatcher processes.

Retry impact: Each retry increases load on both your system and the consumer. Exponential backoff reduces retry rate but increases latency for failed events. For time-sensitive events, you might prefer a fixed retry interval with lower max attempts.

Cost considerations:

• Queue costs: SQS charges per request (10M events = 10M requests, ~$0.40).
• Compute: Running dispatchers on small instances (e.g., t3.micro) can handle thousands of events/sec, costing ~$10/month.
• Network: Outbound data transfer costs apply, but payloads are small.

Security: HMAC signing adds CPU overhead (negligible) but is essential. Use HTTPS to prevent man-in-the-middle attacks.

Observability: Track metrics like delivery success rate, retry counts, and dead-letter queue depth. Use structured logging to correlate event IDs across retries.

Actionable Checklist / Summary

When adopting this pattern in production:

1. Choose a durable queue (SQS, RabbitMQ, etc.) to buffer events.
2. Implement idempotency: Include a unique event ID and require consumers to deduplicate.
3. Use exponential backoff with jitter for retries, and cap the number of attempts.
4. Set up a dead-letter queue for events that fail permanently.
5. Sign payloads with HMAC and validate on the consumer side.
6. Monitor delivery metrics and set up alerts on failure rates.
7. Scale horizontally by running multiple dispatcher instances.
8. Test failure scenarios (consumer down, network partition) to ensure reliability.

By following these guidelines, you can build a webhook delivery system that handles millions of events daily with minimal loss and high reliability.

References

• Webhooks - GitHub Docs
• AWS SQS Documentation
• Exponential Backoff - AWS Architecture Blog
• Webhook Security - Stripe Docs
• Event-Driven Architecture - Microsoft Docs