Skip to content
←Back to Blog
September 3, 2026•6 min read•By Dzaki Amri Zaidaan

Migrating from Monolith to Event‑Driven Microservices with Kafka and Go

Learn how to decompose a monolith into scalable Go microservices using Apache Kafka, implement distributed transactions with the Saga pattern, and achieve eventual consistency in production.

#Backend#Architecture
Migrating from Monolith to Event‑Driven Microservices with Kafka and Go - Digital transformation and generative AI concept

The Problem & Industry Shift

Enterprises are abandoning monolithic codebases because they bottleneck velocity, hinder fault isolation, and amplify blast‑radius during failures. Traditional RPC‑centric microservice migrations still suffer from two‑phase commit limitations—blocking, heavyweight, and unsuitable for cloud‑native elasticity. The industry is converging on event‑driven architectures (EDA) powered by Apache Kafka, which provide durable log‑based communication, natural replayability, and decoupling of service lifecycles. However, moving to EDA introduces new challenges: distributed transaction coordination, eventual consistency guarantees, and operational overhead of managing a streaming platform. This article shows a production‑grade path to address those challenges with Go and Kafka.

Architecture & Core Mechanics

+----------------+        +----------------+        +----------------+
|   API Gateway  |  --->  |   Service A    |  --->  |   Kafka Topic  |
+----------------+        +----------------+        +----------------+
                               |  ^                     |
                               |  |   Saga   Compensate |
                               v  |                     v
+----------------+        +----------------+        +----------------+
|   Service B    |  <---  |   Service C    |  <---  |   Kafka Topic  |
+----------------+        +----------------+        +----------------+
  • Command → Event Flow: A client request hits the API gateway, which forwards a command (e.g., CreateOrder) to Service A. Service A validates the command, persists its local state, and publishes a OrderCreated event to a Kafka topic.
  • Saga Orchestration: Each downstream service (B, C, …) consumes the event, performs its local transaction, and emits a compensating event if it fails. The saga coordinator (often a lightweight state machine inside Service A) tracks progress via a dedicated order-saga topic.
  • Eventual Consistency: Services never block on a global commit. Instead, they converge to a consistent state through idempotent event handling and versioned aggregates.

Key Kafka mechanics

  • Exactly‑once semantics (EOS) with idempotent producers and transactional consumers to avoid duplicate processing.
  • Compact topics for state reconstruction (e.g., order-state topic with key = order‑id).
  • Consumer groups to scale horizontally while preserving ordering per key.

Production Code Example

Below is a minimal, type‑safe Go service that publishes an OrderCreated event using the Sarama client (the de‑facto Kafka library). It demonstrates:

  1. Producer configuration for EOS.
  2. Message keying for partition affinity.
  3. JSON schema enforcement via Go structs.
package main

import (
    "encoding/json"
    "log"
    "os"
    "time"

    "github.com/Shopify/sarama"
)

type OrderCreated struct {
    OrderID   string    `json:"order_id"`
    Customer  string    `json:"customer"`
    Amount    float64   `json:"amount"`
    CreatedAt time.Time `json:"created_at"`
}

func main() {
    // ---- 1️⃣ Producer setup with EOS ----
    cfg := sarama.NewConfig()
    cfg.Producer.Return.Successes = true
    cfg.Producer.Idempotent = true               // enable exactly‑once
    cfg.Producer.RequiredAcks = sarama.WaitForAll // quorum ack
    cfg.Version = sarama.V2_8_0_0                // match broker version

    brokers := []string{"kafka-broker-1:9092", "kafka-broker-2:9092"}
    producer, err := sarama.NewSyncProducer(brokers, cfg)
    if err != nil {
        log.Fatalf("failed to create producer: %v", err)
    }
    defer producer.Close()

    // ---- 2️⃣ Build the event payload ----
    evt := OrderCreated{
        OrderID:   "order-12345",
        Customer:  "alice@example.com",
        Amount:    199.99,
        CreatedAt: time.Now().UTC(),
    }
    payload, err := json.Marshal(evt)
    if err != nil {
        log.Fatalf("json marshal error: %v", err)
    }

    // ---- 3️⃣ Publish with key = OrderID (preserves order per order) ----
    msg := &sarama.ProducerMessage{
        Topic: "order-events",
        Key:   sarama.StringEncoder(evt.OrderID),
        Value: sarama.ByteEncoder(payload),
    }
    partition, offset, err := producer.SendMessage(msg)
    if err != nil {
        log.Fatalf("send message failed: %v", err)
    }
    log.Printf("event persisted to partition %d at offset %d", partition, offset)
}

Why this matters

  • Idempotent producer guarantees that retries (common in flaky networks) do not create duplicate events.
  • Keyed messages ensure all events for a given order land in the same partition, preserving order without extra coordination.
  • SyncProducer is used for simplicity; in high‑throughput services switch to an async producer with a bounded queue and back‑pressure handling.

Performance, Cost & Trade‑offs

DimensionConsiderationTypical Impact
LatencyEOS adds a round‑trip to the leader and acks from replicas.~5‑15 ms per publish on a 3‑node cluster (see Kafka benchmark [1]).
ThroughputBatch size & linger.ms can push >100k msgs/sec per broker.Scale horizontally by adding partitions; each partition caps at ~30k msgs/sec.
Storage CostRetention policy (e.g., 7 days) × replication factor (3) × avg payload (500 B).Roughly 1 GB per 1M events – budget accordingly.
Operational ComplexityRequires Zookeeper/KRaft, monitoring (JMX, Prometheus), and schema registry for compatibility.Higher ops burden vs. simple HTTP RPC, but mitigated by managed services (Confluent Cloud, AWS MSK).
Consistency ModelEventual consistency; compensating actions needed for failures.Simpler than 2PC but requires robust saga design and idempotent consumers.

Trade‑off summary: If your SLA tolerates sub‑second eventual consistency and you need horizontal scalability, Kafka + Saga wins. For strict ACID across services, you still need a distributed transaction manager or redesign to isolate cross‑service invariants.

Actionable Checklist / Summary

  • Define bounded contexts and map each to a Go microservice.
  • Model events as immutable contracts (use protobuf or JSON Schema; store schemas in Confluent Schema Registry).
  • Enable exactly‑once semantics: idempotent producers, transactional consumers, and compacted state topics.
  • Implement saga coordination: either choreography (services emit compensating events) or orchestration (central saga service). Choose based on workflow complexity.
  • Make consumers idempotent: deduplicate using message keys or a Redis/DB dedup table.
  • Set retention & compaction policies to balance replayability vs. storage cost.
  • Instrument with Prometheus metrics (kafka_producer_success_total, consumer_lag) and trace correlation IDs across services.
  • Run chaos tests (e.g., kafka-producer-perf-test.sh with network partitions) to validate resilience.
  • Gradual rollout: start with a thin façade service that forwards monolith calls to Kafka, then incrementally replace monolith modules.

By following this roadmap, teams can transition from a monolith to a resilient, event‑driven microservice ecosystem without sacrificing data integrity.

References

← Explore More ArticlesShare on X / Twitter →