Circuit Breaker Mode Overview
The circuit breaker pattern is used to handle failures in distributed systems, especially in microservice architectures. It prevents cascading failures by detecting failed services and stopping interactions for a specified period of time.
Why and when to use circuit breakers:
- Fail Fast: Stop unnecessary calls to failed services and save system resources.
- Service Stability: Helps maintain overall application responsiveness.
- Recovery: Provides recovery time for failed services.
- Example scenario:
- External services are down.
- Database outage.
- High latency of related services.
Step-by-step implementation in .NET Core 8
Step 1: Install necessary NuGet packages
Install the Polly library:
dotnet add package Polly
dotnet add package Microsoft.Extensions.Http.Polly
Step 2: Set up a resilient HTTP client using circuit breakers
Configure elastic HTTP clients for ProductService, CartService and OrderService.
using Microsoft.Extensions.DependencyInjection;
using Polly;
using Polly.CircuitBreaker;
using Polly.Extensions.Http;
using System;
using System.Net.Http;
var builder = WebApplication.CreateBuilder(args);
// Register HTTP Clients with Circuit Breaker
builder.Services.AddHttpClient("ProductService", client =>
{
client.BaseAddress = new Uri("https://api.example.com/products");
})
.AddPolicyHandler(GetCircuitBreakerPolicy());
builder.Services.AddHttpClient("CartService", client =>
{
client.BaseAddress = new Uri("https://api.example.com/cart");
})
.AddPolicyHandler(GetCircuitBreakerPolicy());
builder.Services.AddHttpClient("OrderService", client =>
{
client.BaseAddress = new Uri("https://api.example.com/orders");
})
.AddPolicyHandler(GetCircuitBreakerPolicy());
var app = builder.Build();
app.Run();
// Circuit Breaker Policy
IAsyncPolicy GetCircuitBreakerPolicy()
{
return HttpPolicyExtensions
.HandleTransientHttpError()
.CircuitBreakerAsync(
handledEventsAllowedBeforeBreaking: 3,
durationOfBreak: TimeSpan.FromSeconds(30),
onBreak: (exception, breakDelay) =>
{
Console.WriteLine($"Circuit broken: {exception.Message}");
},
onReset: () =>
{
Console.WriteLine("Circuit reset");
},
onHalfOpen: () =>
{
Console.WriteLine("Circuit in half-open state");
});
}
Step 3: Create a service layer to call HTTP clients
Encapsulate HTTP client logic in a service.
public class ProductService
{
private readonly HttpClient _httpClient;
public ProductService(IHttpClientFactory httpClientFactory)
{
_httpClient = httpClientFactory.CreateClient("ProductService");
}
public async Task GetProductsAsync()
{
var response = await _httpClient.GetAsync("/list");
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
public class CartService
{
private readonly HttpClient _httpClient;
public CartService(IHttpClientFactory httpClientFactory)
{
_httpClient = httpClientFactory.CreateClient("CartService");
}
public async Task GetCartDetailsAsync()
{
var response = await _httpClient.GetAsync("/details");
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
public class OrderService
{
private readonly HttpClient _httpClient;
public OrderService(IHttpClientFactory httpClientFactory)
{
_httpClient = httpClientFactory.CreateClient("OrderService");
}
public async Task GetOrderDetailsAsync()
{
var response = await _httpClient.GetAsync("/details");
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
Step 4: Register the service in dependency injection
Add the service class to the dependency injection container.
builder.Services.AddScoped();
builder.Services.AddScoped();
builder.Services.AddScoped();
Step 5: Use the service in the controller
Use the service in the controller.
using Microsoft.AspNetCore.Mvc;
[ApiController]
[Route("api/[controller]")]
public class ShopController : ControllerBase
{
private readonly ProductService _productService;
private readonly CartService _cartService;
private readonly OrderService _orderService;
public ShopController(ProductService productService, CartService cartService, OrderService orderService)
{
_productService = productService;
_cartService = cartService;
_orderService = orderService;
}
[HttpGet("products")]
public async Task GetProducts()
{
var products = await _productService.GetProductsAsync();
return Ok(products);
}
[HttpGet("cart")]
public async Task GetCart()
{
var cartDetails = await _cartService.GetCartDetailsAsync();
return Ok(cartDetails);
}
[HttpGet("orders")]
public async Task GetOrders()
{
var orderDetails = await _orderService.GetOrderDetailsAsync();
return Ok(orderDetails);
}
}
best practices
- 1. Isolating circuit breaker: Use separate circuit breakers for each service.
To avoid cascading failures and ensure independent business processing, use separate circuit breakers for each business.
- Use named HTTP clients with unique circuit breaker policies for each service.
- Avoid sharing circuit breaker policies across multiple services.
builder.Services.AddHttpClient("ProductService", client =>
{
client.BaseAddress = new Uri("https://api.example.com/products");
})
.AddPolicyHandler(GetProductServiceCircuitBreakerPolicy());
builder.Services.AddHttpClient("CartService", client =>
{
client.BaseAddress = new Uri("https://api.example.com/cart");
})
.AddPolicyHandler(GetCartServiceCircuitBreakerPolicy());
builder.Services.AddHttpClient("OrderService", client =>
{
client.BaseAddress = new Uri("https://api.example.com/orders");
})
.AddPolicyHandler(GetOrderServiceCircuitBreakerPolicy());
IAsyncPolicy GetProductServiceCircuitBreakerPolicy() =>
HttpPolicyExtensions
.HandleTransientHttpError()
.CircuitBreakerAsync(3, TimeSpan.FromSeconds(30));
IAsyncPolicy GetCartServiceCircuitBreakerPolicy() =>
HttpPolicyExtensions
.HandleTransientHttpError()
.CircuitBreakerAsync(2, TimeSpan.FromSeconds(20));
IAsyncPolicy GetOrderServiceCircuitBreakerPolicy() =>
HttpPolicyExtensions
.HandleTransientHttpError()
.CircuitBreakerAsync(5, TimeSpan.FromSeconds(60));
-
2. Monitoring and logging: Continuously monitor circuit breaker events for debugging and alerting. Monitors events such as circuit breaks, resets, and transitions to the half-open state. This helps with debugging and tracing the health of the service.
-
Use onBreak, onReset, and onHalfOpen callbacks for logging.
-
Leverage monitoring tools like Application Insights or ELK Stack for observability.
IAsyncPolicy GetCircuitBreakerPolicyWithLogging()
{
return HttpPolicyExtensions
.HandleTransientHttpError()
.CircuitBreakerAsync(
handledEventsAllowedBeforeBreaking: 3,
durationOfBreak: TimeSpan.FromSeconds(30),
onBreak: (exception, timespan) =>
{
Console.WriteLine($"Circuit broken due to: {exception.Message}. Duration: {timespan}");
// Log to Application Insights or other tools
},
onReset: () =>
{
Console.WriteLine("Circuit reset to closed state.");
},
onHalfOpen: () =>
{
Console.WriteLine("Circuit in half-open state. Testing health...");
});
}
-
3. Elegant fallback: Provides meaningful fallback responses when services are unavailable. This enhances the user experience and ensures the application remains functional.
-
Use Polly’s fallback strategy with a circuit breaker.
-
Return cached or default data.
IAsyncPolicy GetCircuitBreakerWithFallback()
{
return Policy.WrapAsync(
Policy
.Handle()
.FallbackAsync(new HttpResponseMessage(System.Net.HttpStatusCode.OK)
{
Content = new StringContent("Service temporarily unavailable. Using fallback.")
}),
HttpPolicyExtensions
.HandleTransientHttpError()
.CircuitBreakerAsync(3, TimeSpan.FromSeconds(30))
);
}
4. Graceful downgrade and function switching
Use feature switches to gracefully degrade functionality when critical services fail. Tools like LaunchDarkly or FeatureManagement in Azure Application Settings can help.
5. Use centralized configuration
Store circuit breaker configurations in a centralized location for easy updates and management. Tools like Azure App Configuration or Consul are perfect for this purpose.
public class CircuitBreakerSettings
{
public int HandledEventsAllowedBeforeBreaking { get; set; }
public int DurationOfBreakInSeconds { get; set; }
}
builder.Services.Configure(
builder.Configuration.GetSection("CircuitBreaker"));
IAsyncPolicy GetCircuitBreakerPolicyWithConfig(IOptionsMonitor options)
{
var settings = options.CurrentValue;
return HttpPolicyExtensions
.HandleTransientHttpError()
.CircuitBreakerAsync(
settings.HandledEventsAllowedBeforeBreaking,
TimeSpan.FromSeconds(settings.DurationOfBreakInSeconds));
}
Monitoring and testing tools
Application insights:
Used to log circuit breaker events and track service health.
Azure Application Insights
ELK stack:
ElasticSearch, Logstash and Kibana for centralized logging.
Gremlins:
- Chaos engineering tools for simulating service failures.
- gremlin
Postman/Newman:
- For manual or automatic testing of APIs under failure conditions.
- postman
Polly Dashboard:
- Community-based dashboard for monitoring Polly policies.
- Polly DashboardGitHub
online resources
Polly Documentation: Polly official website
Circuit breaker design pattern: Microsoft Learning
Microservices best practices: Microservices on .NET