Building Resilient Microservices with Circuit Breakers

The Cascading Failure Problem

In a microservices architecture, services call each other. When Service B is slow, Service A's threads block waiting for responses. If enough threads block, Service A's thread pool exhausts. A's latency spikes. Now A's callers (Services C and D) also block. Within minutes, a single degraded downstream service has cascaded into a system-wide outage.

This is not a theoretical scenario. It is the most common cause of full-system failures in distributed architectures. The fix is not adding more timeouts — it is proactive fault isolation via circuit breakers.

How Circuit Breakers Work

The circuit breaker pattern, named after electrical breakers, monitors calls to a remote service. It transitions through three states:

Closed (normal operation): Calls flow through. The breaker tracks the error rate over a sliding window.

Open (faulting): When errors exceed a threshold (e.g., 50% failure in the last 10 calls), the breaker opens. Subsequent calls fail immediately without reaching the downstream service. This gives the failing service time to recover without being hammered by retries.

Half-open (probing): After a cooldown period, the breaker allows a small number of test calls through. If they succeed, the breaker closes. If they fail, it reopens.

Closed → (failure threshold exceeded) → Open
Open → (cooldown expires) → Half-Open
Half-Open → (probe succeeds) → Closed
Half-Open → (probe fails) → Open

Implementation in Node.js

A minimal circuit breaker is ~60 lines of JavaScript:

type CircuitState = 'closed' | 'open' | 'half-open';

interface CircuitBreakerConfig {
  failureThreshold: number;   // % failures to trip breaker
  successThreshold: number;   // successes in half-open to close
  timeout: number;            // ms to wait before half-open probe
  volumeThreshold: number;    // min requests before evaluating
}

class CircuitBreaker {
  private state: CircuitState = 'closed';
  private failures = 0;
  private successes = 0;
  private requests = 0;
  private lastFailureTime = 0;

  constructor(
    private fn: (...args: unknown[]) => Promise<unknown>,
    private config: CircuitBreakerConfig
  ) {}

  async call(...args: unknown[]): Promise<unknown> {
    if (this.state === 'open') {
      const elapsed = Date.now() - this.lastFailureTime;
      if (elapsed < this.config.timeout) {
        throw new Error('Circuit breaker OPEN — call rejected');
      }
      this.state = 'half-open';
    }

    try {
      const result = await this.fn(...args);
      this.onSuccess();
      return result;
    } catch (err) {
      this.onFailure();
      throw err;
    }
  }

  private onSuccess(): void {
    this.failures = 0;
    if (this.state === 'half-open') {
      this.successes++;
      if (this.successes >= this.config.successThreshold) {
        this.state = 'closed';
        this.successes = 0;
      }
    }
  }

  private onFailure(): void {
    this.failures++;
    this.requests++;
    this.lastFailureTime = Date.now();
    
    if (
      this.requests >= this.config.volumeThreshold &&
      (this.failures / this.requests) >= (this.config.failureThreshold / 100)
    ) {
      this.state = 'open';
    }
  }
}

Usage:

const paymentBreaker = new CircuitBreaker(
  (orderId: string) => paymentService.charge(orderId),
  {
    failureThreshold: 50,
    successThreshold: 2,
    timeout: 10_000,
    volumeThreshold: 5
  }
);

// In your request handler:
try {
  await paymentBreaker.call(order.id);
} catch (err) {
  if (err.message.includes('OPEN')) {
    // Breaker is open — use fallback behavior
    await queueForRetry(order);
  } else {
    throw err; // Real error from downstream
  }
}

Implementation in Go

Go's concurrency primitives make thread-safe circuit breakers straightforward:

type CircuitBreaker struct {
    mu           sync.RWMutex
    state        string
    failures     int
    lastFailure  time.Time
    threshold    int
    timeout      time.Duration
}

func NewCircuitBreaker(threshold int, timeout time.Duration) *CircuitBreaker {
    return &CircuitBreaker{
        state:     "closed",
        threshold: threshold,
        timeout:   timeout,
    }
}

func (cb *CircuitBreaker) Call(fn func() error) error {
    cb.mu.RLock()
    state := cb.state
    lastFailure := cb.lastFailure
    cb.mu.RUnlock()

    if state == "open" {
        if time.Since(lastFailure) < cb.timeout {
            return fmt.Errorf("circuit breaker open")
        }
        cb.mu.Lock()
        cb.state = "half-open"
        cb.mu.Unlock()
    }

    err := fn()
    
    cb.mu.Lock()
    defer cb.mu.Unlock()
    
    if err != nil {
        cb.failures++
        cb.lastFailure = time.Now()
        if cb.failures >= cb.threshold {
            cb.state = "open"
        }
        return err
    }
    
    cb.failures = 0
    cb.state = "closed"
    return nil
}

Choosing Thresholds

The right thresholds depend on your traffic patterns and SLAs:

Failure threshold (50% default): Too low and noisy services trip constantly. Too high and you let failures propagate too long. Start at 50%, measure, adjust.

Timeout (10-30s default): Shorter timeouts mean faster recovery attempts but more failed probes. Match this to your downstream service's typical restart time.

Volume threshold (5-20 requests): Without this, one failed request in a low-traffic service trips the breaker. Wait for statistical significance.

Sliding window size: Small windows (last 10 requests) respond faster but are noisier. Large windows (last 60 seconds) are more stable but slower to react.

Production Considerations

Expose state via health endpoints: Circuit breaker state is critical operational information:

app.get('/health', (req, res) => {
  res.json({
    status: 'ok',
    circuitBreakers: {
      payment: paymentBreaker.getState(),
      inventory: inventoryBreaker.getState(),
    }
  });
});

Metric emission: Emit metrics on state transitions. Alert when any breaker opens. Track time-in-open-state per service.

Fallback strategies: Define what happens when a breaker is open. Options: return cached data, return a degraded response, queue for async processing, or fail fast with a meaningful error. Never silently swallow the error.

Bulkheads: Combine circuit breakers with bulkheads (isolated thread pools per dependency). A slow downstream service then can only exhaust its own pool, not the global pool.

Using Libraries in Production

For production use, prefer battle-tested libraries over rolling your own:

• Node.js: opossum (maintained, well-tested, Prometheus metrics built-in)
• Go: sony/gobreaker (simple, idiomatic)
• Java: Resilience4j (replaces Hystrix, Spring Boot integration)
• Service mesh: Istio and Linkerd implement circuit breaking at the proxy layer, language-agnostic

The circuit breaker pattern does not eliminate failures. It contains them, making the difference between a single service degradation and a system-wide outage. That containment is often the difference between a 30-minute incident and a 4-hour one.