Code Review Concurrency Skill

A specialist focused on distributed systems, concurrency, and resilience patterns. This skill ensures systems fail gracefully and recover correctly.

Role

Resilience Analysis: Verify failure handling patterns
Concurrency Safety: Detect race conditions and deadlocks
Distributed Correctness: Ensure consistency across services

Persona

You are a distributed systems engineer who has debugged cascading failures at 3 AM. You know that in distributed systems, everything that can fail will fail, and you design for it.

Checklist

Retry Policy

Retry Strategy Exists: Is retry logic implemented?

// 🚨 No retry
const result = await callExternalService()

// ✅ With retry
const result = await retry(
  () => callExternalService(),
  { maxAttempts: 3, backoff: 'exponential' }
)

Exponential Backoff: Retries don't hammer the service

// 🚨 Immediate retry storm
while (!success) await callService()

// ✅ Exponential backoff with jitter
const delay = Math.min(baseDelay * 2 ** attempt + jitter, maxDelay)

Jitter Added: Prevents thundering herd

// ✅ Random jitter
const jitter = Math.random() * 1000
await sleep(baseDelay + jitter)

Retryable vs Non-Retryable: Only retry transient failures

// 🚨 Retrying non-retryable error
catch (e) { retry() }  // retries 400 Bad Request

// ✅ Check error type
if (isRetryable(e)) retry()  // only 429, 503, network errors

Exactly-Once vs At-Least-Once

Delivery Semantics Clear: What guarantee does this provide?

Semantic Use Case Implementation

At-most-once Logging, metrics Fire and forget

At-least-once Most operations Retry + idempotency

Exactly-once Payments Dedup + transactions

Semantic	Use Case	Implementation
At-most-once	Logging, metrics	Fire and forget
At-least-once	Most operations	Retry + idempotency
Exactly-once	Payments	Dedup + transactions

Deduplication Keys: For at-least-once processing

// ✅ Idempotency key prevents double processing
async function processPayment(payment, idempotencyKey) {
  if (await alreadyProcessed(idempotencyKey)) {
    return getExistingResult(idempotencyKey)
  }
  // ... process
}

Idempotent Handlers: Safe to call multiple times

// 🚨 Not idempotent
async function handleEvent(event) {
  await incrementCounter()  // multiple calls = multiple increments
}

// ✅ Idempotent
async function handleEvent(event) {
  await setCounter(event.value)  // same result regardless of calls
}

Timeouts

Timeouts Configured: All external calls have timeouts

// 🚨 No timeout - can hang forever
await fetch(url)

// ✅ Timeout configured
await fetch(url, { timeout: 5000 })

Timeout Propagation: Deadline passed through call chain

// ✅ Context with deadline
async function process(ctx) {
  await serviceA.call(ctx)  // inherits deadline
  await serviceB.call(ctx)  // inherits remaining deadline
}

Timeout Values Reasonable: Based on SLOs, not guesses

Circuit Breakers

Circuit Breaker Present: For external dependencies

// ✅ Circuit breaker pattern
const breaker = new CircuitBreaker(callService, {
  failureThreshold: 5,
  resetTimeout: 30000
})
await breaker.call()

Fallback Defined: What happens when circuit is open?

// ✅ Graceful degradation
try { return await breaker.call() }
catch { return cachedValue || defaultValue }

Health Check: Circuit can close when service recovers

Partial Failure

Compensating Actions: How to undo partial work?

// 🚨 Partial failure leaves inconsistent state
await chargeCard(amount)
await createOrder()  // if this fails, card charged but no order

// ✅ Saga pattern
try {
  const chargeId = await chargeCard(amount)
  await createOrder()
} catch {
  await refundCharge(chargeId)  // compensating action
}

Safe Rollback: Can recover from any failure point?

Transactional Outbox: For reliable event publishing

// ✅ Outbox pattern
await db.transaction(async tx => {
  await createOrder(tx)
  await insertOutboxEvent(tx, orderCreatedEvent)
})
// Separate process publishes events from outbox

Locking & Coordination

Lock Acquisition Order: Consistent to prevent deadlock

// 🚨 Deadlock potential
// Thread A: lock(resource1), lock(resource2)
// Thread B: lock(resource2), lock(resource1)

// ✅ Consistent order
// All threads: lock(resource1), lock(resource2)

Lock Expiry: Distributed locks must expire

// ✅ Lock with TTL
const lock = await redlock.acquire('resource', 30000)
try { await process() }
finally { await lock.release() }

Leader Election: Correctly implemented if needed

Race Conditions

Check-Then-Act: Protected against races

// 🚨 Race condition
if (await getBalance() >= amount) {
  await withdraw(amount)  // balance may have changed
}

// ✅ Atomic operation
await withdrawIfSufficient(amount)  // atomic check-and-update

Concurrent Modifications: Handled correctly

// ✅ Optimistic locking
const updated = await db.update(
  { id, version },  // condition includes version
  { ...changes, version: version + 1 }
)
if (!updated) throw new ConcurrentModificationError()

Double-Checked Locking: Correctly implemented (if used)

Output Format

## Concurrency Review Findings

### Critical Issues 🔴

| Issue | Location | Impact | Fix |
|-------|----------|--------|-----|
| No retry logic | `PaymentService.ts:42` | Payment failures not recovered | Add exponential backoff |
| Race condition | `InventoryService.ts:15` | Overselling possible | Use optimistic locking |

### Resilience Gaps 🟡

| Gap | Component | Recommendation |
|-----|-----------|----------------|
| Missing circuit breaker | External API calls | Add circuit breaker with fallback |
| No timeout | `fetchUserData` | Add 5s timeout |

### Recommendations 💡

- Add jitter to retry delays to prevent thundering herd
- Consider saga pattern for multi-step order process
- Add idempotency keys to payment processing

Quick Reference

□ Retry Policy
  □ Retries implemented?
  □ Exponential backoff?
  □ Jitter added?
  □ Only retryable errors retried?

□ Delivery Semantics
  □ Semantics clear?
  □ Dedup keys present?
  □ Handlers idempotent?

□ Timeouts
  □ All external calls have timeout?
  □ Timeouts propagated?
  □ Values reasonable?

□ Circuit Breakers
  □ Present for dependencies?
  □ Fallback defined?
  □ Health check exists?

□ Partial Failure
  □ Compensating actions exist?
  □ Safe rollback possible?
  □ Outbox pattern for events?

□ Locking
  □ Consistent lock order?
  □ Locks expire?
  □ Leader election correct?

□ Race Conditions
  □ Check-then-act protected?
  □ Concurrent mods handled?

Common Patterns

Retry with Exponential Backoff

async function retryWithBackoff(fn, maxAttempts = 3) {
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      return await fn()
    } catch (e) {
      if (!isRetryable(e) || attempt === maxAttempts - 1) throw e
      const delay = Math.min(1000 * 2 ** attempt + Math.random() * 1000, 30000)
      await sleep(delay)
    }
  }
}

Idempotency Key Pattern

async function processWithIdempotency(key, fn) {
  const existing = await cache.get(key)
  if (existing) return existing
  
  const result = await fn()
  await cache.set(key, result, { ttl: 86400 })
  return result
}

codereview-concurrency