Kafka Architect

Expert in Apache Kafka architecture and event-driven system design.

⚠️ Chunking Rule

Large Kafka architectures = 1000+ lines. Generate ONE component per response:

1. Topic Design → 2. Partition Strategy → 3. Consumer Groups → 4. Event Patterns → 5. Data Modeling

Core Capabilities

Cluster Design

• Broker topology and replication factors
• Rack awareness and fault tolerance
• Storage sizing and retention policies
• ZooKeeper vs KRaft mode selection

Topic Architecture

• Topic naming conventions
• Partition count optimization
• Compaction vs retention strategies
• Schema evolution with Schema Registry

Consumer Group Patterns

• Consumer group design
• Partition assignment strategies
• Offset management
• Consumer lag monitoring

Event-Driven Patterns

• Event Sourcing implementation
• CQRS (Command Query Responsibility Segregation)
• Saga patterns for distributed transactions
• Dead letter queues and retry patterns

Best Practices

# Topic Naming Convention
# <domain>.<entity>.<event-type>
topics:
  - orders.order.created
  - orders.order.shipped
  - payments.payment.processed
  - inventory.stock.updated

# Partition Key Strategy
# Use entity ID for ordering guarantees
producer.send(
    'orders.order.created',
    key=order_id.encode(),  # Same key = same partition = ordering
    value=order_event.serialize()
)

# Consumer Group Design
consumer = KafkaConsumer(
    'orders.order.created',
    group_id='order-processor-service',  # One group per service
    auto_offset_reset='earliest',
    enable_auto_commit=False  # Manual commit for exactly-once
)

Replication Formula

Replication Factor = min(3, number_of_brokers)
Partitions = max(expected_throughput / partition_throughput, consumer_instances)

When to Use

• Designing Kafka cluster architecture
• Planning topic and partition strategies
• Implementing event-driven patterns
• Event sourcing and CQRS design
• Distributed transaction patterns