Designing Resilient API Layers for Production

The optimistic API is the standard. You write a function that calls a remote service, await the response, and handle the result. If the call fails, an error propagates up. The caller decides what to do.

This model is adequate when failures are rare and brief. In production distributed systems, failures are frequent and their duration is unpredictable. A single downstream service having a bad 30 seconds can cascade into minutes of degraded user experience if the API layer is not designed for it.

Resilient API design does not prevent failures. It prevents failures from cascading, minimizes their blast radius, and ensures the system degrades gracefully rather than catastrophically.

The Circuit Breaker

The circuit breaker is the most important resilience pattern and the one most frequently absent. Its purpose: detect when a downstream service is failing and stop sending requests to it before the failure cascades.

A circuit breaker has three states:

• Closed — requests flow normally; failures are counted
• Open — requests fail immediately without reaching the downstream service; the circuit "tripped"
• Half-open — a single probe request is allowed through; if it succeeds, the circuit closes; if it fails, it reopens

class CircuitBreaker {
  private failures   = 0
  private lastFailed = 0
  private state: 'closed' | 'open' | 'half-open' = 'closed'

  constructor(
    private readonly threshold:   number = 5,
    private readonly timeout:     number = 30_000,
    private readonly halfOpenDelay: number = 5_000,
  ) {}

  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === 'open') {
      if (Date.now() - this.lastFailed > this.timeout) {
        this.state = 'half-open'
      } else {
        throw new Error('Circuit open — downstream unavailable')
      }
    }

    try {
      const result = await fn()
      this.onSuccess()
      return result
    } catch (err) {
      this.onFailure()
      throw err
    }
  }

  private onSuccess() {
    this.failures = 0
    this.state    = 'closed'
  }

  private onFailure() {
    this.failures++
    this.lastFailed = Date.now()
    if (this.failures >= this.threshold) {
      this.state = 'open'
    }
  }
}

The key property: when the circuit is open, requests fail immediately with a known error rather than waiting for the downstream timeout. If your downstream times out after 10 seconds and you receive 100 requests per second, an open circuit saves 1,000 seconds of accumulated thread-blocking per second of downstream unavailability.

Retry With Exponential Backoff and Jitter

Retry is the first instinct for transient failures, and it is correct — with caveats. Naive retry (immediate, on every error) converts a momentary blip into a thundering herd. Every client retries simultaneously, overwhelming the service that just became available.

Exponential backoff with jitter distributes retry load across time:

async function withRetry<T>(
  fn: () => Promise<T>,
  options: { maxAttempts?: number; baseDelay?: number; maxDelay?: number } = {},
): Promise<T> {
  const { maxAttempts = 3, baseDelay = 100, maxDelay = 10_000 } = options
  let lastError: unknown

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn()
    } catch (err) {
      lastError = err
      if (attempt === maxAttempts) break
      if (!isRetryable(err)) break   // 4xx errors are not retried

      // Exponential backoff: 100ms, 200ms, 400ms...
      // Jitter: ±30% of the delay to distribute load
      const exp   = baseDelay * 2 ** (attempt - 1)
      const delay = Math.min(exp, maxDelay)
      const jitter = delay * 0.3 * (Math.random() * 2 - 1)
      await sleep(delay + jitter)
    }
  }

  throw lastError
}

function isRetryable(err: unknown): boolean {
  if (err instanceof Response) {
    // Server errors are retryable; client errors are not
    return err.status >= 500
  }
  if (err instanceof Error) {
    // Network errors are retryable
    return err.name === 'NetworkError' || err.name === 'AbortError'
  }
  return false
}

Timeout Hierarchies

Every external call must have a timeout. This is not optional — it is the difference between a slow failure and an indefinite hang that exhausts your connection pool.

Timeouts should be hierarchical: the outer operation's timeout must be shorter than the sum of all inner timeouts. Otherwise an inner retry-after-failure can outlast the outer operation's deadline, wasting the retry effort:

// Total operation budget: 5 seconds
const OPERATION_TIMEOUT = 5_000

// Individual call timeout: 1.5 seconds
// With 2 retries: max 4.5 seconds (within budget)
const CALL_TIMEOUT = 1_500

async function fetchWithTimeout<T>(
  url: string,
  timeout: number = CALL_TIMEOUT,
): Promise<T> {
  const controller = new AbortController()
  const timer      = setTimeout(() => controller.abort(), timeout)
  try {
    const res = await fetch(url, { signal: controller.signal })
    if (!res.ok) throw new Error(`HTTP ${res.status}`)
    return res.json()
  } finally {
    clearTimeout(timer)
  }
}

Bulkhead Isolation

Bulkheads prevent a single misbehaving integration from consuming all available concurrency. The name comes from ship design: compartmentalized hulls where flooding one compartment does not flood the ship.

In API terms: if you call ten different downstream services, each service gets a fixed concurrency budget. If Service A becomes slow and backs up, it cannot consume the concurrency that Service B needs:

class Bulkhead {
  private active = 0

  constructor(private readonly limit: number) {}

  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.active >= this.limit) {
      throw new Error(`Bulkhead limit reached (${this.limit} concurrent)`)
    }
    this.active++
    try {
      return await fn()
    } finally {
      this.active--
    }
  }
}

const githubBulkhead = new Bulkhead(10)   // max 10 concurrent GitHub calls
const supabaseBulkhead = new Bulkhead(20)  // max 20 concurrent Supabase calls

// Calls through different bulkheads cannot starve each other
await githubBulkhead.call(() => fetchGitHubStats(repo))
await supabaseBulkhead.call(() => fetchUserData(userId))

Graceful Degradation — The Most Important Pattern

All of the above patterns prevent failures from cascading. Graceful degradation determines what the user experiences when a failure cannot be prevented.

The design principle: every feature that depends on an external service must have a degraded mode that is still useful. Not "an error message" — a degraded but functional state.

Concrete examples:

• GitHub stats fail → show cached stats from last successful fetch (with a "last updated" indicator)
• Search service fails → show "Search temporarily unavailable, try again shortly" rather than a blank search results page
• Analytics collection fails → silently discard; the user experience is unaffected
• Recommended content fails → show most recent content instead of personalized content

async function getGitHubStats(repo: string): Promise<GitHubStats> {
  try {
    const live = await githubBreaker.call(() =>
      withRetry(() => fetchGitHubStats(repo), { maxAttempts: 2 })
    )
    await cache.set(`gh:${repo}`, live, { ttl: 3600 })
    return live
  } catch (err) {
    reportError(err, { context: 'getGitHubStats', repo })
    // Degraded mode: serve stale cache with a staleness indicator
    const cached = await cache.get<GitHubStats>(`gh:${repo}`)
    if (cached) return { ...cached, stale: true }
    // Last resort: static fallback from public/data/github.json
    return loadStaticGitHubData(repo)
  }
}

The degraded state serves the user. The error is logged. The circuit breaker tracks the failure. The system continues operating at reduced fidelity rather than failing completely.

Composing the Patterns

The patterns compose: a circuit breaker wraps a retry, which wraps a bulkhead-gated call with a timeout. The composition determines the system's behavior under different failure modes:

const breaker  = new CircuitBreaker(5, 30_000)
const bulkhead = new Bulkhead(10)

async function resilientFetch<T>(url: string): Promise<T> {
  return breaker.call(() =>
    bulkhead.call(() =>
      withRetry(() => fetchWithTimeout<T>(url), { maxAttempts: 2 })
    )
  )
}

This function: limits concurrency (bulkhead), times out slow calls, retries transient failures, and stops sending requests if the service is consistently failing (circuit breaker). Five patterns composed into one function. The caller gets reliability without complexity.

Resilient API design is infrastructure engineering applied at the application layer. It does not prevent the failures that distributed systems guarantee — it determines whether those failures are experienced by users or absorbed by the system. In production, that distinction matters more than almost any feature you can ship.