Kafka Topology Design Skill

Design deterministic, replay-safe, cost-efficient Kafka Streams topologies using KSA patterns.

Required Reading

Before responding, load the shared reference:

cat ${SKILL_PATH}/references/KSA.md

This is the authoritative source for all patterns, principles, and constraints.

---

Workflow

Step 1 — Understand the Problem

Gather from the user (ask if missing):

1. Source events — what topics trigger this service?
2. Enrichment needs — what data beyond the event itself?
3. Outputs — output topics, DB writes, notifications?
4. Statefulness — does output depend on past events?
5. Partition key — what entity scopes this? (userId, orderId, etc.)

Step 2 — Select Recipes

Map to KSA recipes (KSA.md §4):

Problem involves…	Recipe
Cleaning/validating inbound events	01 — Validation & Normalization
Duplicate events from upstream	02 — Deduplication
Splitting events to different consumers	03 — Routing & Fan-Out
Looking up reference data	04 — Data Enrichment
Reference data + historical computation	05 — Enrichment + Stateful
Counting, rate limiting, windowed metrics	06 — Windowed Aggregation
Entity lifecycle (order, payment, KYC)	07 — Per-Key State Machine
Cross-service coordination with rollback	08 — Saga Orchestrator
Building a read model or search index	09 — CQRS Projection
Bug fix replay or data backfill	10 — Event Replay

Step 3 — Compose the Topology

Arrange recipes left to right:

Source → [Ingress] → [Enrichment] → [Computation] → [Egress] → Sink

Not every stage needed. Only include what the problem requires.

Step 4 — Draw the Diagram

Produce a Mermaid flowchart LR using the KSA symbol legend (KSA.md §3):

• [TopicName] — Kafka topic
• [TopicName*] — compacted topic (KTable source)
• (Processor) — stateless processor
• {{Processor}} — stateful processor
• ((Join)) — stream–table join
• [[Sink]] — side-effect boundary
• {Decision?} — conditional branch

Step 5 — Declare Policies

For every topology, explicitly document:

1. Missing-state policy per join (drop / dead-letter / retry / buffer)
2. Partition key and why it aligns with all joins
3. State store retention per stateful processor
4. Sink idempotency strategy

Step 6 — Cost Check

Estimate per KSA.md §7.4:

Factor	Estimate	Red Flag
State store size/key	value × keys × retention	> 50 GB/instance
Changelog overhead	store size × replication	> 100 GB total
Repartition count	selectKey/through calls	> 2 on high-volume
KTable restore time	topic size / throughput	> 10 minutes
Partition count	all internal + output topics	> 500 total

Multiple red flags → recommend alternatives (KSA.md §7.3).

Step 7 — Compliance Checklist

Verify against KSA.md §6 before signing off.

---

Output Format

Always produce:

1. Summary — one paragraph describing what the service does
2. Recipes used — numbered list of KSA recipe numbers and names
3. Topology diagram — Mermaid flowchart LR
4. State diagram — Mermaid stateDiagram-v2 (if FSM involved)
5. Policy table — missing-state, retention, idempotency decisions
6. Cost estimate — back-of-napkin numbers for the heuristic
7. Compliance — checklist pass/fail

---

Anti-Patterns to Flag

Anti-Pattern	Why It's Wrong	Suggest
HTTP calls inside processor	Breaks replay determinism	KTable enrichment
DB queries inside processor	Same as above	Compacted topic
No declared missing-state policy	Undeclared behavior = design defect	Ask: "what happens when KTable has no entry?"
Partition key mismatch	Join key ≠ partition key	Repartition (flag cost)
Unbounded state stores	No TTL = unbounded growth	Ask about retention
GlobalKTable for large data	Loads ALL data on EVERY instance	Regular KTable with partition-aligned joins
Multiple repartitions on same stream	Each doubles I/O	Redesign key strategy
Stateful where stateless suffices	Unnecessary state store overhead	Remove state store

---

Conversation Style

• Ask clarifying questions before designing. Problem statements are often incomplete.
• When multiple recipes apply, explain trade-offs and let the engineer choose.
• Always produce a diagram.
• Be explicit about what the topology does NOT handle (scope).
• If the problem is better solved without KStreams, say so (KSA.md §7.3).