Building Resilient Systems: Error Handling and Recovery Patterns

Why resilience matters in distributed systems

Failures are inevitable in distributed systems. The difference between a good system and a great one is how gracefully they handle failures. Here's how to build resilient systems that recover automatically.

Failure Modes

Common Failures

1. Network Failures
   • Timeouts
   • Connection refused
   • Packet loss
2. Service Failures
   • Service crashes
   • High latency
   • Resource exhaustion
3. Database Failures
   • Connection pool exhaustion
   • Query timeouts
   • Deadlocks
4. Third-Party Failures
   • API outages
   • Rate limiting
   • Invalid responses

Resilience Patterns

1Retry Pattern

Many failures in a distributed system are brief: a dropped connection, a service restarting, a moment of overload. The retry pattern repeats a failed call a few times before giving up, so a short blip never reaches the user. The skill is in retrying well: waiting a little longer after each attempt (exponential backoff), adding random jitter so many clients don't retry at the same instant, and stopping after a set number of tries. Only retry operations that are safe to repeat, and never retry errors that will fail the same way every time, such as invalid input.

When to Retry

• Transient failures
• Network issues
• Temporary service unavailability

Implementation

async function retryOperation(operation, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await operation();
    } catch (error) {
      if (i === maxRetries - 1) throw error;
      await sleep(Math.pow(2, i) * 1000); // Exponential backoff
    }
  }
}

Best Practices

• Exponential backoff
• Jitter to prevent thundering herd
• Maximum retry limit
• Don't retry non-retryable errors

2Circuit Breaker Pattern

Purpose

Prevent cascading failures by stopping requests to failing services.

States

• Closed: Normal operation
• Open: Failing, reject requests immediately
• Half-Open: Testing if service recovered

Implementation

class CircuitBreaker {
  constructor(threshold = 5, timeout = 60000) {
    this.failureCount = 0;
    this.threshold = threshold;
    this.timeout = timeout;
    this.state = 'CLOSED';
    this.nextAttempt = Date.now();
  }

  async execute(operation) {
    if (this.state === 'OPEN') {
      if (Date.now() < this.nextAttempt) {
        throw new Error('Circuit breaker is OPEN');
      }
      this.state = 'HALF_OPEN';
    }

    try {
      const result = await operation();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  onSuccess() {
    this.failureCount = 0;
    this.state = 'CLOSED';
  }

  onFailure() {
    this.failureCount++;
    if (this.failureCount >= this.threshold) {
      this.state = 'OPEN';
      this.nextAttempt = Date.now() + this.timeout;
    }
  }
}

3Bulkhead Pattern

Purpose

Isolate resources to prevent total system failure.

Example

• Separate thread pools
• Isolated database connections
• Separate service instances

Benefits

• Failure isolation
• Resource protection
• Better resource utilization

4Timeout Pattern

Always Set Timeouts

const controller = new AbortController();
const timeoutId = setTimeout(() => controller.abort(), 5000);

try {
  const response = await fetch(url, {
    signal: controller.signal
  });
} finally {
  clearTimeout(timeoutId);
}

5Fallback Pattern

Provide Alternatives

async function getData() {
  try {
    return await primaryService.getData();
  } catch (error) {
    console.warn('Primary service failed, using fallback');
    return await fallbackService.getData();
  }
}

Fallback Options

• Cached data
• Default values
• Alternative service
• Degraded functionality

6Health Checks

Monitor Service Health

app.get('/health', async (req, res) => {
  const checks = {
    database: await checkDatabase(),
    redis: await checkRedis(),
    externalApi: await checkExternalApi()
  };

  const healthy = Object.values(checks).every(c => c === true);
  res.status(healthy ? 200 : 503).json({ checks });
});

Error Handling Best Practices

1Structured Error Handling

Error Types

class AppError extends Error {
  constructor(message, code, statusCode) {
    super(message);
    this.code = code;
    this.statusCode = statusCode;
  }
}

class ValidationError extends AppError {
  constructor(message) {
    super(message, 'VALIDATION_ERROR', 400);
  }
}

2Error Logging

Log Everything

• Error message
• Stack trace
• Context (user, request, etc.)
• Timestamp

Tools

• Sentry
• LogRocket
• Datadog
• Custom logging

3User-Friendly Error Messages

Don't Expose Internals

// Bad
res.status(500).json({ error: error.stack });

// Good
res.status(500).json({ 
  error: 'Something went wrong. Please try again later.' 
});

4Graceful Degradation

Degrade Features, Not Entire App

• Show cached data
• Disable non-critical features
• Show maintenance message
• Allow core functionality

Monitoring and Alerting

Key Metrics

• Error rates
• Response times
• Circuit breaker states
• Retry counts
• Timeout rates

Alerting

Alert On

• High error rates
• Circuit breaker opens
• Service degradation
• Unusual patterns

Testing Resilience

Chaos Engineering

Purpose

Test system behavior under failure conditions.

Techniques

• Kill services randomly
• Inject latency
• Simulate network failures
• Resource exhaustion

Tools

• Chaos Monkey
• Litmus
• Gremlin

Where to start with resilience

Building resilient systems requires anticipating failures and implementing patterns to handle them gracefully. Start with retries and timeouts, add circuit breakers for critical services, implement health checks, and continuously monitor. Resilience is not optional—it's essential.

Need help building resilient systems? [Contact us](/schedule-appointment) for expert guidance.