[Question]: How to use a CircuitBreaker(basic or advanced) policy with multiple retry policies

Question

[Question]: How to use a CircuitBreaker(basic or advanced) policy with multiple retry policies

Mahantesh-DotNet opened this issue 6 months ago · 17 comments

What are you wanting to achieve?

I have attached my usecase or question in the attached document. Please help either in polly 7 or 8 version. Thanks in advance.

What code or approach do you have so far?

I have attached the code snippet in the attachment.

Additional context

No response

Answer 1 · 2024-07-05T17:54:42.000Z

There is no attachment.

Answer 2 · 2024-07-05T17:58:24.000Z

///

/// Retry these codes for 3 times for every second
///

private static readonly List _set1 =
[
HttpStatusCode.Unauthorized //401
];

///

/// Policy to retry for 3 times for every second
///

public AsyncRetryPolicy WaitAndRetrySet1Async()
{
//jitter strategy with ConstantBackoff
var constantBackoff = Backoff.ConstantBackoff(delay: TimeSpan.FromSeconds(1), retryCount: 3);

AsyncRetryPolicy<HttpResponseMessage> waitAndRetryPolicy = Policy
    .HandleResult<HttpResponseMessage>(r => _set1.Contains(r.StatusCode))
    .Or<BrokenCircuitException>()
    .WaitAndRetryAsync(sleepDurations: constantBackoff, onRetry: OnRetry1);
return waitAndRetryPolicy;

}

///

/// Retry these codes for 3 times exponentially 1,2,4
///

private static readonly List _set2 =
[
HttpStatusCode.TooManyRequests //429
];
///

/// Policy to retry for 3 times for 1,2,4 seconds
///

public AsyncRetryPolicy WaitAndRetrySet2Async ()
{
//jitter strategy with an ExponentialBackoff
var exponentialBackoff = Backoff.ExponentialBackoff(initialDelay: TimeSpan.FromSeconds(1), retryCount: 3);

AsyncRetryPolicy<HttpResponseMessage> waitAndRetryPolicy = Policy
    .HandleResult<HttpResponseMessage>(r => _set2.Contains(r.StatusCode))
    .Or<BrokenCircuitException>()
    .WaitAndRetryAsync(sleepDurations: exponentialBackoff, onRetry: OnRetry2);
return waitAndRetryPolicy;

}

///

/// Retry these codes for 3 times every 5 second
///

private static readonly List _set3 =
[
HttpStatusCode.BadGateway, //502
HttpStatusCode.GatewayTimeout //504
];
///

/// Policy to retry for 3 times for every 5 second
///

public AsyncRetryPolicy WaitAndRetrySet3Async()
{
//jitter strategy with ConstantBackoff
var constantBackoff = Backoff.ConstantBackoff(delay: TimeSpan.FromSeconds(5), retryCount: 3);

AsyncRetryPolicy<HttpResponseMessage> waitAndRetryPolicy = Policy
    .HandleResult<HttpResponseMessage>(r => _set3.Contains(r.StatusCode))
    .Or<BrokenCircuitException>()
    .WaitAndRetryAsync(sleepDurations: constantBackoff, onRetry: OnRetry3);
return waitAndRetryPolicy;

}

///

/// This is the circuit breaker policy to work with the above set of all status codes _set1, _set2, _set3.
/// That is a common circuit breaker policy which should work with a set of status codes.
///

public AsyncCircuitBreakerPolicy BasicCircuitBreakerForAllStatusCodesAsync()
{
AsyncCircuitBreakerPolicy basicCircuitBreakerPolicy = Policy
.HandleResult(r => _set1.Contains(r.StatusCode) || _set2.Contains(r.StatusCode) || _set3.Contains(r.StatusCode))
.CircuitBreakerAsync(handledEventsAllowedBeforeBreaking: 2,
durationOfBreak: TimeSpan.FromSeconds(10),
onBreak: OnBreak, onReset: OnReset, onHalfOpen: OnHalfOpen);
return basicCircuitBreakerPolicy;
}
I am applying all these policies to my RestApi as below
builder.Services.AddHttpClient(“TestClient”, client =>
{
client.BaseAddress = new Uri(http://localhost:5000);
})
.AddPolicyHandler(retryPolicies.WaitAndRetrySet1Async())
.AddPolicyHandler(retryPolicies.WaitAndRetrySet2Async())
.AddPolicyHandler(retryPolicies.WaitAndRetrySet3Async())
.AddPolicyHandler(retryPolicies.BasicCircuitBreakerForAllStatusCodesAsync());

My use case below:

I will submit 2 requests at a time
GetEmployeeByName I expect 429 response here
GetEmployeeById I expect 502 response here

In this case, GetEmployeeByName and GetEmployeeById requests will be submitted. These requests will fail as expected(intended to fail for testing purpose). Now circuit will transition to OPEN state for 10 seconds, after 10 seconds circuit will transition to Half-Open and allow either GetEmployeeByName or GetEmployeeById to process. Once all these retry requests completed. After sometime again I will submit the same requests and I should be able to see the same behaviour. But I am not able to see the desired output. May I know whether I am missing something or can you please share me any piece of example code to achieve this. Thanks in advance.

Note: Retry policies are created in one class and circuit breaker policy in another class.

Answer 3 · 2024-07-08T07:02:28.000Z

@Mahantesh-DotNet Please allow me to reformat your shared code fragment to make it more legible

Retry 1

///
/// Retry these codes for 3 times for every second
///
private static readonly List _set1 = 
[
  HttpStatusCode.Unauthorized //401
];

///
/// Policy to retry for 3 times for every second
///
public AsyncRetryPolicy WaitAndRetrySet1Async() 
{
  //jitter strategy with ConstantBackoff
  var constantBackoff = Backoff.ConstantBackoff(delay: TimeSpan.FromSeconds(1), retryCount: 3);

  AsyncRetryPolicy < HttpResponseMessage > waitAndRetryPolicy = Policy
    .HandleResult < HttpResponseMessage > (r => _set1.Contains(r.StatusCode))
    .Or < BrokenCircuitException > ()
    .WaitAndRetryAsync(sleepDurations: constantBackoff, onRetry: OnRetry1);
  return waitAndRetryPolicy;

}

Retry 2

///
/// Retry these codes for 3 times exponentially 1,2,4
///
private static readonly List _set2 = 
[
  HttpStatusCode.TooManyRequests //429
];

///
/// Policy to retry for 3 times for 1,2,4 seconds
///
public AsyncRetryPolicy WaitAndRetrySet2Async() 
{
  //jitter strategy with an ExponentialBackoff
  var exponentialBackoff = Backoff.ExponentialBackoff(initialDelay: TimeSpan.FromSeconds(1), retryCount: 3);

  AsyncRetryPolicy < HttpResponseMessage > waitAndRetryPolicy = Policy
    .HandleResult < HttpResponseMessage > (r => _set2.Contains(r.StatusCode))
    .Or < BrokenCircuitException > ()
    .WaitAndRetryAsync(sleepDurations: exponentialBackoff, onRetry: OnRetry2);
  return waitAndRetryPolicy;

}

Retry 3

///
/// Retry these codes for 3 times every 5 second
///
private static readonly List _set3 = 
[
  HttpStatusCode.BadGateway, //502
  HttpStatusCode.GatewayTimeout //504
];

///
/// Policy to retry for 3 times for every 5 second
///
public AsyncRetryPolicy WaitAndRetrySet3Async() 
{
  //jitter strategy with ConstantBackoff
  var constantBackoff = Backoff.ConstantBackoff(delay: TimeSpan.FromSeconds(5), retryCount: 3);

  AsyncRetryPolicy < HttpResponseMessage > waitAndRetryPolicy = Policy
    .HandleResult < HttpResponseMessage > (r => _set3.Contains(r.StatusCode))
    .Or < BrokenCircuitException > ()
    .WaitAndRetryAsync(sleepDurations: constantBackoff, onRetry: OnRetry3);
  return waitAndRetryPolicy;

}

Circuit Breaker

///
/// This is the circuit breaker policy to work with the above set of all status codes _set1, _set2, _set3.
/// That is a common circuit breaker policy which should work with a set of status codes.
///
public AsyncCircuitBreakerPolicy BasicCircuitBreakerForAllStatusCodesAsync() 
{
  AsyncCircuitBreakerPolicy basicCircuitBreakerPolicy = Policy
    .HandleResult(r => _set1.Contains(r.StatusCode) || _set2.Contains(r.StatusCode) || _set3.Contains(r.StatusCode))
    .CircuitBreakerAsync(handledEventsAllowedBeforeBreaking: 2,
      durationOfBreak: TimeSpan.FromSeconds(10),
      onBreak: OnBreak, onReset: OnReset, onHalfOpen: OnHalfOpen);
  return basicCircuitBreakerPolicy;
}

I am applying all these policies to my RestApi as below

builder.Services.AddHttpClient(“TestClient”, client => 
    {
         client.BaseAddress = new Uri(http: //localhost:5000);
    })
    .AddPolicyHandler(retryPolicies.WaitAndRetrySet1Async())
    .AddPolicyHandler(retryPolicies.WaitAndRetrySet2Async())
    .AddPolicyHandler(retryPolicies.WaitAndRetrySet3Async())
    .AddPolicyHandler(retryPolicies.BasicCircuitBreakerForAllStatusCodesAsync());

After sometime again I will submit the same requests and I should be able to see the same behaviour. But I am not able to see the desired output.

Could you please describe the observed and the expected behavior?

Answer 4 · 2024-07-08T11:03:16.000Z

I am sharing the updated code(corrected with logs)

Answer 5 · 2024-07-08T11:04:16.000Z

using Polly.CircuitBreaker;
using Polly.Contrib.WaitAndRetry;
using Polly.Retry;
using Polly;
using System.Net;

namespace ClientAPI.TestHarness.Modules
{
public class WaitAndRetryPolicies
{

    ### Retry1

    /// <summary>
    /// Retry these codes for 3 times for every second
    /// </summary>
    private static readonly List<HttpStatusCode> _set1 =
    [
        HttpStatusCode.Unauthorized     //401
    ];

    /// <summary>
    /// Policy to retry for 3 times for every second
    /// </summary>
    public AsyncRetryPolicy<HttpResponseMessage> WaitAndRetrySet1Async()
    {
        //jitter strategy with ConstantBackoff
        var constantBackoff = Backoff.ConstantBackoff(delay: TimeSpan.FromSeconds(1), retryCount: 3);

        AsyncRetryPolicy<HttpResponseMessage> waitAndRetryPolicy = Policy
            .HandleResult<HttpResponseMessage>(r => _set1.Contains(r.StatusCode))
            .Or<BrokenCircuitException>()
            .WaitAndRetryAsync(sleepDurations: constantBackoff, onRetry: OnRetryWaitAndRetrySet1);
        return waitAndRetryPolicy;
    }

    private void OnRetryWaitAndRetrySet1(DelegateResult<HttpResponseMessage> result, TimeSpan timeSpan, int retryCount, Context ctx)
    {
        Console.WriteLine($"{DateTime.Now.ToString("HH:mm:ss.fff")} {nameof(OnRetryWaitAndRetrySet1)} Retry# {retryCount} will be executed after {timeSpan.TotalSeconds} seconds");
    }

    ### Retry2

    /// <summary>
    /// Retry these codes for 3 times exponentially 1,2,4
    /// </summary>
    private static readonly List<HttpStatusCode> _set2 =
    [
        HttpStatusCode.TooManyRequests      //429
    ];

    /// <summary>
    /// Policy to retry for 3 times for 1,2,4 seconds
    /// </summary>
    public AsyncRetryPolicy<HttpResponseMessage> WaitAndRetrySet2Async()
    {
        //jitter strategy with an ExponentialBackoff
        var exponentialBackoff = Backoff.ExponentialBackoff(initialDelay: TimeSpan.FromSeconds(1), retryCount: 3);

        AsyncRetryPolicy<HttpResponseMessage> waitAndRetryPolicy = Policy
            .HandleResult<HttpResponseMessage>(r => _set2.Contains(r.StatusCode))
            .Or<BrokenCircuitException>()
            .WaitAndRetryAsync(sleepDurations: exponentialBackoff, onRetry: OnRetryWaitAndRetrySet2);
        return waitAndRetryPolicy;
    }

    private void OnRetryWaitAndRetrySet2(DelegateResult<HttpResponseMessage> result, TimeSpan timeSpan, int retryCount, Context ctx)
    {
        Console.WriteLine($"{DateTime.Now.ToString("HH:mm:ss.fff")} {nameof(OnRetryWaitAndRetrySet2)} Retry# {retryCount} will be executed after {timeSpan.TotalSeconds} seconds");
    }

    ### Retry3

    /// <summary>
    /// Retry these codes for 3 times every 5 second
    /// </summary>
    private static readonly List<HttpStatusCode> _set3 =
    [
        HttpStatusCode.BadGateway,           	//502
        HttpStatusCode.GatewayTimeout           //504
    ];

    /// <summary>
    /// Policy to retry for 3 times for every 5 second
    /// </summary>
    public AsyncRetryPolicy<HttpResponseMessage> WaitAndRetrySet3Async()
    {
        //jitter strategy with ConstantBackoff
        var constantBackoff = Backoff.ConstantBackoff(delay: TimeSpan.FromSeconds(5), retryCount: 3);

        AsyncRetryPolicy<HttpResponseMessage> waitAndRetryPolicy = Policy
            .HandleResult<HttpResponseMessage>(r => _set3.Contains(r.StatusCode))
            .Or<BrokenCircuitException>()
            .WaitAndRetryAsync(sleepDurations: constantBackoff, onRetry: OnRetryWaitAndRetrySet3);
        return waitAndRetryPolicy;
    }

    private void OnRetryWaitAndRetrySet3(DelegateResult<HttpResponseMessage> result, TimeSpan timeSpan, int retryCount, Context ctx)
    {
        Console.WriteLine($"{DateTime.Now.ToString("HH:mm:ss.fff")} {nameof(OnRetryWaitAndRetrySet3)} Retry# {retryCount} will be executed after {timeSpan.TotalSeconds} seconds");
    }
}

}

Answer 6 · 2024-07-08T11:05:31.000Z

using Polly.CircuitBreaker;
using Polly;
using System.Net;

namespace ClientAPI.TestHarness.Modules
{
public class CircuitBreakerPolicy
{

    ### Circuit breaker
    /// <summary>
    /// Retry these codes for 3 times for every second
    /// </summary>
    private static readonly List<HttpStatusCode> _set1 =
    [
        HttpStatusCode.Unauthorized     //401
    ];

    /// <summary>
    /// Retry these codes for 3 times exponentially 1,2,4
    /// </summary>
    private static readonly List<HttpStatusCode> _set2 =
    [
        HttpStatusCode.TooManyRequests      //429
    ];

    /// <summary>
    /// Retry these codes for 3 times every 5 second
    /// </summary>
    private static readonly List<HttpStatusCode> _set3 =
    [
        HttpStatusCode.BadGateway,           	//502
        HttpStatusCode.GatewayTimeout           //504
    ];

    /// <summary>
    /// This is the circuit breaker policy to work with the above set of all status codes _set1, _set2, _set3.
    /// That is a common circuit breaker policy which should work with a set of status codes.
    /// </summary>
    public AsyncCircuitBreakerPolicy<HttpResponseMessage> CommonCircuitBreakerForAllStatusCodesAsync()
    {
        AsyncCircuitBreakerPolicy<HttpResponseMessage> basicCircuitBreakerPolicy = Policy
            .HandleResult<HttpResponseMessage>(r => _set1.Contains(r.StatusCode) || _set2.Contains(r.StatusCode) || _set3.Contains(r.StatusCode))
            .CircuitBreakerAsync(handledEventsAllowedBeforeBreaking: 2,
                durationOfBreak: TimeSpan.FromSeconds(10),
                onBreak: OnBreak, onReset: OnReset, onHalfOpen: OnHalfOpen);
        return basicCircuitBreakerPolicy;
    }

    private void OnBreak(DelegateResult<HttpResponseMessage> result, TimeSpan timeSpan)
    {
        Console.WriteLine($"{DateTime.Now.ToString("HH:mm:ss.fff")} {nameof(OnBreak)} Circuit transitioned to OPEN state, Duration of break is {timeSpan.TotalSeconds} seconds ");
    }

    private void OnReset()
    {
        Console.WriteLine($"{DateTime.Now.ToString("HH:mm:ss.fff")} {nameof(OnReset)} Circuit transitioned to CLOSED state");
    }

    private void OnHalfOpen()
    {
        Console.WriteLine($"{DateTime.Now.ToString("HH:mm:ss.fff")} {nameof(OnHalfOpen)} Circuit transitioned to Half-Open state");
    }
}

}

Answer 7 · 2024-07-08T11:07:04.000Z

  Hosting starting

info: Microsoft.Hosting.Lifetime[14]
Now listening on: http://localhost:5025
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
Hosting environment: Development
dbug: Microsoft.Extensions.Hosting.Internal.Host[2]
Hosting started
dbug: Microsoft.WebTools.BrowserLink.Net.BrowserLinkMiddleware[1]
Response markup is scheduled to include Browser Link script injection.
dbug: Microsoft.WebTools.BrowserLink.Net.BrowserLinkMiddleware[2]
Response markup was updated to include Browser Link script injection.
dbug: ClientAPI.TestHarness.Controllers[0]
Begin-Constructor-TestController
dbug: ClientAPI.TestHarness.Controllers[0]
End-Constructor-TestController
dbug: ClientAPI.TestHarness.Controllers[0]
[Begin] TestController.HttpListner
info: ClientAPI.TestHarness.Controllers[0]
14:48:31.241- Executing the request from user- 502
info: System.Net.Http.HttpClient.HttpListener6002.LogicalHandler[100]
Start processing HTTP request GET http://localhost:6002/502
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100]
Sending HTTP request GET http://localhost:6002/502
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101]
Received HTTP response headers after 253.9542ms - 502
14:48:31.571 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds
dbug: ClientAPI.TestHarness.Controllers[0]
Begin-Constructor-TestController
dbug: ClientAPI.TestHarness.Controllers[0]
End-Constructor-TestController
dbug: ClientAPI.TestHarness.Controllers[0]
[Begin] TestController.HttpListner6002
info: ClientAPI.TestHarness.Controllers[0]
14:48:31.953- Executing the request from user- 401
info: System.Net.Http.HttpClient.HttpListener6002.LogicalHandler[100]
Start processing HTTP request GET http://localhost:6002/401
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100]
Sending HTTP request GET http://localhost:6002/401
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101]
Received HTTP response headers after 122.093ms - 401
14:48:32.127 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
(mycomment-CORRECT behaviour, since both the user requests are failed, the circuit transitioned to OPEN state
because handledEventsAllowedBeforeBreaking=2 for circuit breaker)

14:48:32.128 OnRetryWaitAndRetrySet1 Retry# 1 will be executed after 1 seconds
14:48:33.141 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds
14:48:36.588 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds
14:48:38.155 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds
14:48:41.598 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds
14:48:43.161 OnHalfOpen Circuit transitioned to Half-Open state
(mycomment-CORRECT behaviour, circuit was in OPEN state and transitioned to Half-Open state because Duration of break is 10 seconds)

info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100]
Sending HTTP request GET http://localhost:6002/401
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101]
Received HTTP response headers after 45.0781ms - 401
(mycomment-may be an incorrect behaviour, since the circuit was in OPEN state from 14:48:32 to 14:48:42, transitioned to half open state at 14:48:43 the retry request#1, #2, #3 of the retry policy(WaitAndRetrySet1Async) to the url http://localhost:6002/401 should have been attempted at 14:48:33, 14:48:34, 14:48:35 and expect to fail since the circuit was in OPEN state and no request was allowed to flow.
On other hand, the other parallel request http://localhost:6002/502 should have been retried by the policy (WaitAndRetrySet3Async) for 2 times at 14:48:36(retry #1) and 14:48:41(retry #2) and expect these requests to fail since the circuit was in OPEN state and no request was allowed to flow)

14:48:43.216 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
(mycomment-CORRECT behaviour, circuit transitioned from half-open to OPEN state since the last request was failed)

14:48:43.216 OnRetryWaitAndRetrySet1 Retry# 2 will be executed after 1 seconds
14:48:44.220 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds
14:48:46.613 OnRetryWaitAndRetrySet2 Retry# 1 will be executed after 1 seconds
14:48:47.620 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds
14:48:49.227 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds
14:48:52.636 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds

(mycomment-may be an incorrect behaviour since this log should not be printed(OnRetryWaitAndRetrySet2) as I have not recieved any response which belongs to OnRetryWaitAndRetrySet2 method)

(mycomment-may be an incorrect behaviour, since the circuit was in OPEN state from 14:48:43 to 14:48:53, transitioned to half open state at 14:48:54 the retry request #3 & #4 should have been retried at 14:48:46 and at 14:48:51 or 52 by the policy (WaitAndRetrySet3Async) to the url http://localhost:6002/502 and is exptected to fail since circuit was in OPEN state and no request was allowed to flow)

14:48:54.227 OnHalfOpen Circuit transitioned to Half-Open state
(mycomment-CORRECT behaviour, circuit was in OPEN state and transitioned to Half-Open state because Duration of break is 10 seconds)

info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100]
Sending HTTP request GET http://localhost:6002/401
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101]
Received HTTP response headers after 35.1538ms - 401

14:48:54.268 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
(mycomment-CORRECT behaviour, circuit transitioned from half-open to OPEN state since the last request was failed)

14:48:54.269 OnRetryWaitAndRetrySet1 Retry# 3 will be executed after 1 seconds
14:48:55.277 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds
14:48:57.647 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds
14:49:00.291 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds
14:49:02.660 OnRetryWaitAndRetrySet2 Retry# 2 will be executed after 2 seconds
14:49:04.668 OnHalfOpen Circuit transitioned to Half-Open state
(mycomment-CORRECT behaviour, circuit was in OPEN state and transitioned to Half-Open state because Duration of break is 10 seconds)

(mycomment-may be an incorrect behaviour, the retry request #5 should have been retried at 14:48:56 and is exptected to fail since circuit was in OPEN state and no request was allowed to flow)

info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100]
Sending HTTP request GET http://localhost:6002/502
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101]
Received HTTP response headers after 51.3886ms - 502
14:49:04.729 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
(mycomment-CORRECT behaviour, circuit transitioned from half-open to OPEN state since the last request was failed)

14:49:04.729 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds
14:49:05.312 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds
14:49:09.745 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds
14:49:10.327 OnRetryWaitAndRetrySet2 Retry# 1 will be executed after 1 seconds
14:49:11.341 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds
14:49:14.754 OnHalfOpen Circuit transitioned to Half-Open state
(mycomment-CORRECT behaviour, circuit was in OPEN state and transitioned to Half-Open state because Duration of break is 10 seconds)

info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100]
Sending HTTP request GET http://localhost:6002/502
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101]
Received HTTP response headers after 33.7128ms - 502
14:49:14.795 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
14:49:14.795 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds
14:49:16.343 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds
14:49:19.802 OnRetryWaitAndRetrySet2 Retry# 3 will be executed after 4 seconds
14:49:21.354 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds
14:49:23.811 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds
14:49:26.360 OnHalfOpen Circuit transitioned to Half-Open state
(mycomment-CORRECT behaviour, circuit was in OPEN state and transitioned to Half-Open state because Duration of break is 10 seconds)

info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100]
Sending HTTP request GET http://localhost:6002/401
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101]
Received HTTP response headers after 50.1859ms - 401
14:49:26.416 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
info: System.Net.Http.HttpClient.HttpListener6002.LogicalHandler[101]
End processing HTTP request after 54460.8472ms - 401
info: ClientAPI.TestHarness.Controllers[0]
14:49:26.423 [End] TestController.HttpListner6002- The service responded with 401-Unauthorized status code for the request URI http://localhost:6002/401. All the retries are completed.
dbug: ClientAPI.TestHarness.Controllers[0]
14:49:26- [End] TestController.HttpListner6002
14:49:28.829 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds
14:49:33.853 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds
14:49:38.858 OnHalfOpen Circuit transitioned to Half-Open state
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100]
Sending HTTP request GET http://localhost:6002/502
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101]
Received HTTP response headers after 37.823ms - 502
14:49:38.904 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
info: System.Net.Http.HttpClient.HttpListener6002.LogicalHandler[101]
End processing HTTP request after 67644.8797ms - 502
info: ClientAPI.TestHarness.Controllers[0]
14:49:38.908 [End] TestController.HttpListner- The service responded with 502-Bad Gateway status code for the request URI http://localhost:6002/502. All the retries are completed.
dbug: ClientAPI.TestHarness.Controllers[0]
14:49:38- [End] TestController.HttpListner

======================2nd time submitted the same requests again============================

  dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101]
  Ending HttpMessageHandler cleanup cycle after 0.005ms - processed: 0 items - remaining: 1 items

dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100]
Starting HttpMessageHandler cleanup cycle with 1 items
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101]
Ending HttpMessageHandler cleanup cycle after 0.0094ms - processed: 0 items - remaining: 1 items
dbug: ClientAPI.TestHarness.Controllers[0]
Begin-Constructor-TestController
dbug: ClientAPI.TestHarness.Controllers[0]
End-Constructor-TestController
dbug: ClientAPI.TestHarness.Controllers[0]
[Begin] TestController.HttpListner
info: ClientAPI.TestHarness.Controllers[0]
14:54:06.173- Executing the request from user- 502
info: System.Net.Http.HttpClient.HttpListener6002.LogicalHandler[100]
Start processing HTTP request GET http://localhost:6002/502
14:54:06.187 OnHalfOpen Circuit transitioned to Half-Open state
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100]
Sending HTTP request GET http://localhost:6002/502
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101]
Received HTTP response headers after 138.088ms - 502
14:54:06.340 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
(mycomment-CORRECT behaviour, circuit transitioned from half-open to OPEN state since the last request was failed)

14:54:06.341 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds
dbug: ClientAPI.TestHarness.Controllers[0]
Begin-Constructor-TestController
dbug: ClientAPI.TestHarness.Controllers[0]
End-Constructor-TestController
dbug: ClientAPI.TestHarness.Controllers[0]
[Begin] TestController.HttpListner6002
info: ClientAPI.TestHarness.Controllers[0]
14:54:06.913- Executing the request from user- 401
info: System.Net.Http.HttpClient.HttpListener6002.LogicalHandler[100]
Start processing HTTP request GET http://localhost:6002/401
14:54:06.920 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds
14:54:11.332 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100]
Starting HttpMessageHandler cleanup cycle with 1 items
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101]
Ending HttpMessageHandler cleanup cycle after 0.0063ms - processed: 0 items - remaining: 1 items
14:54:11.920 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds
14:54:16.340 OnHalfOpen Circuit transitioned to Half-Open state
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100]
Sending HTTP request GET http://localhost:6002/502
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101]
Received HTTP response headers after 44.898ms - 502
14:54:16.395 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
14:54:16.395 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds
14:54:16.920 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds
14:54:21.408 OnRetryWaitAndRetrySet2 Retry# 1 will be executed after 1 seconds
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100]
Starting HttpMessageHandler cleanup cycle with 1 items
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101]
Ending HttpMessageHandler cleanup cycle after 0.005ms - processed: 0 items - remaining: 1 items
14:54:21.925 OnRetryWaitAndRetrySet2 Retry# 1 will be executed after 1 seconds
14:54:22.422 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds
14:54:22.928 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds
14:54:27.427 OnHalfOpen Circuit transitioned to Half-Open state
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100]
Sending HTTP request GET http://localhost:6002/502
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101]
Received HTTP response headers after 50.187ms - 502
14:54:27.486 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
14:54:27.486 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds
14:54:27.936 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100]
Starting HttpMessageHandler cleanup cycle with 1 items
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101]
Ending HttpMessageHandler cleanup cycle after 0.0057ms - processed: 0 items - remaining: 1 items
14:54:32.499 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds
14:54:32.935 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds
14:54:37.499 OnHalfOpen Circuit transitioned to Half-Open state
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100]
Sending HTTP request GET http://localhost:6002/502
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101]
Received HTTP response headers after 36.1713ms - 502
14:54:37.541 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
info: System.Net.Http.HttpClient.HttpListener6002.LogicalHandler[101]
End processing HTTP request after 31387.3779ms - 502
info: ClientAPI.TestHarness.Controllers[0]
14:54:37.544 [End] TestController.HttpListner- The service responded with 502-Bad Gateway status code for the request URI http://localhost:6002/502. All the retries are completed.
dbug: ClientAPI.TestHarness.Controllers[0]
14:54:37- [End] TestController.HttpListner
14:54:37.942 OnRetryWaitAndRetrySet2 Retry# 2 will be executed after 2 seconds
14:54:39.957 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100]
Starting HttpMessageHandler cleanup cycle with 1 items
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101]
Ending HttpMessageHandler cleanup cycle after 0.0507ms - processed: 0 items - remaining: 1 items
14:54:44.970 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds
14:54:49.981 OnHalfOpen Circuit transitioned to Half-Open state
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100]
Sending HTTP request GET http://localhost:6002/401
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101]
Received HTTP response headers after 193.4952ms - 401
14:54:50.329 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
14:54:50.329 OnRetryWaitAndRetrySet1 Retry# 1 will be executed after 1 seconds
14:54:51.329 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100]
Starting HttpMessageHandler cleanup cycle with 1 items
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101]
Ending HttpMessageHandler cleanup cycle after 0.0041ms - processed: 0 items - remaining: 1 items
14:54:56.343 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds
14:55:01.338 OnHalfOpen Circuit transitioned to Half-Open state
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100]
Sending HTTP request GET http://localhost:6002/401
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101]
Received HTTP response headers after 40.0117ms - 401
14:55:01.384 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
14:55:01.385 OnRetryWaitAndRetrySet1 Retry# 2 will be executed after 1 seconds
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100]
Starting HttpMessageHandler cleanup cycle with 1 items
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101]
Ending HttpMessageHandler cleanup cycle after 0.0043ms - processed: 0 items - remaining: 1 items
14:55:02.400 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds
14:55:07.406 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100]
Starting HttpMessageHandler cleanup cycle with 1 items
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101]
Ending HttpMessageHandler cleanup cycle after 0.0053ms - processed: 0 items - remaining: 1 items
14:55:12.408 OnHalfOpen Circuit transitioned to Half-Open state
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100]
Sending HTTP request GET http://localhost:6002/401
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101]
Received HTTP response headers after 32.8548ms - 401
14:55:12.445 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
14:55:12.446 OnRetryWaitAndRetrySet1 Retry# 3 will be executed after 1 seconds
14:55:13.454 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds
14:55:18.452 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100]
Starting HttpMessageHandler cleanup cycle with 1 items
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101]
Ending HttpMessageHandler cleanup cycle after 0.0255ms - processed: 0 items - remaining: 1 items
14:55:23.461 OnHalfOpen Circuit transitioned to Half-Open state
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100]
Sending HTTP request GET http://localhost:6002/401
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101]
Received HTTP response headers after 59.3668ms - 401
14:55:23.528 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
info: System.Net.Http.HttpClient.HttpListener6002.LogicalHandler[101]
End processing HTTP request after 76684.2992ms - 401
info: ClientAPI.TestHarness.Controllers[0]
14:55:23.564 [End] TestController.HttpListner6002- The service responded with 401-Unauthorized status code for the request URI http://localhost:6002/401. All the retries are completed.
dbug: ClientAPI.TestHarness.Controllers[0]
14:55:23- [End] TestController.HttpListner6002
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100]
Starting HttpMessageHandler cleanup cycle with 1 items
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101]
Ending HttpMessageHandler cleanup cycle after 0.0042ms - processed: 0 items - remaining: 1 items
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100]
Starting HttpMessageHandler cleanup cycle with 1 items
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101]
Ending HttpMessageHandler cleanup cycle after 0.0052ms - processed: 0 items - remaining: 1 items

Answer 8 · 2024-07-08T11:08:48.000Z

I have written some comments(mycomment-) in the log. Can you please review it.

Answer 9 · 2024-07-08T11:21:36.000Z

This is how I applied the policies at startup.cs

WaitAndRetryPolicies waitAndRetryPolicies = new WaitAndRetryPolicies();
CircuitBreakerPolicy circuitBreakerPolicy = new CircuitBreakerPolicy();

var retry1 = waitAndRetryPolicies.WaitAndRetrySet1Async();
var retry2 = waitAndRetryPolicies.WaitAndRetrySet2Async();
var retry3 = waitAndRetryPolicies.WaitAndRetrySet3Async();

var allRetries = Policy.WrapAsync(retry1, retry2, retry3);
var commonCircuitBreaker = allRetries.WrapAsync(circuitBreakerPolicy.CommonCircuitBreakerForAllStatusCodesAsync());

//Creating the named client instance for HttpListener6002
builder.Services.AddHttpClient("HttpListener6002", client =>
{
client.BaseAddress = new Uri(http://localhost:6002/); //BaseAddress for named client instance HttpListener6002
})
.AddPolicyHandler(commonCircuitBreaker);

Answer 10 · 2024-07-08T11:58:24.000Z

This is my problem statement

Define policies for the following use cases
1)Status code 401 retry for 3 times for every second
2)Status code 423 retry for 3 times exponentially 1,2,4 seconds
3)Status codes 502/504 retry for 3 times for every 5 second

I would like to define the retry policies for the above 3 use cases and I would like to define a common circuit breaker policy which should work for all these 3 use cases for multiple simultaneous requests for a single named http client instance.

Answer 11 · 2024-07-08T11:59:21.000Z

Can you please correct me if anything is wrong in my code or please point me to right solution to achieve the same. Thank you in advance.

Answer 12 · 2024-07-09T07:58:25.000Z

@Mahantesh-DotNet No, your implementation does not work correctly. Let me demonstrate it.

Here is a simplified version of your code:

public static async Task Main()
{
	var strategy = Policy.WrapAsync<HttpResponseMessage>(RetryScenario1(), RetryScenario2(), RetryScenario3(), CircuitBreaker());
	await strategy.ExecuteAsync(() => Task.FromResult(new HttpResponseMessage(HttpStatusCode.Unauthorized)));
}

static IAsyncPolicy<HttpResponseMessage> RetryScenario1() 
	=> Policy<HttpResponseMessage>
	.HandleResult(r => r.StatusCode == HttpStatusCode.Unauthorized)
	.Or<BrokenCircuitException>()
	.WaitAndRetryAsync(sleepDurations: Backoff.ConstantBackoff(delay: TimeSpan.FromSeconds(1), retryCount: 3), 
			   onRetry: (ex, ts) => Console.WriteLine("OnRetry for 401"));

static IAsyncPolicy<HttpResponseMessage> RetryScenario2() 
	=> Policy<HttpResponseMessage>
	.HandleResult(r => r.StatusCode == HttpStatusCode.TooManyRequests)
	.Or<BrokenCircuitException>()
	.WaitAndRetryAsync(sleepDurations: Backoff.ExponentialBackoff(initialDelay: TimeSpan.FromSeconds(1), retryCount: 3), 
			   onRetry: (ex, ts) => Console.WriteLine("OnRetry for 429"));

static IAsyncPolicy<HttpResponseMessage> RetryScenario3() 	
	=> Policy<HttpResponseMessage>
	.HandleResult(r => r.StatusCode == HttpStatusCode.BadGateway || r.StatusCode == HttpStatusCode.GatewayTimeout)
	.Or<BrokenCircuitException>()
	.WaitAndRetryAsync(sleepDurations: Backoff.ConstantBackoff(delay: TimeSpan.FromSeconds(5), retryCount: 3), 
			   onRetry: (ex, ts) => Console.WriteLine("OnRetry for 502, 504"));

static List<HttpStatusCode> retryable = [ HttpStatusCode.Unauthorized, HttpStatusCode.TooManyRequests, HttpStatusCode.BadGateway, HttpStatusCode.GatewayTimeout ];
static IAsyncPolicy<HttpResponseMessage> CircuitBreaker() 
	=> Policy<HttpResponseMessage>
	.HandleResult(r => retryable.Contains(r.StatusCode))
	.CircuitBreakerAsync(handledEventsAllowedBeforeBreaking: 2,
			     durationOfBreak: TimeSpan.FromSeconds(10),
			     onBreak: (r, ts) => Console.WriteLine("OnBreak"), 
			     onReset: () => Console.WriteLine("OnReset"), 
			     onHalfOpen: () => Console.WriteLine("OnHalfOpen"));

If I run this code inside dotnet fiddle then I receive the following output:

OnRetry for 401
OnBreak
OnRetry for 401
OnRetry for 502, 504
OnRetry for 502, 504

So, why do we have OnRetry for 502, 504 if we always return 401?

Well the reason is that your retry policies triggers for BrokenCircuitException as well. So, when the CB is open then it shortcuts the execution by throwing a BrokenCircuitException. The next policy in the policy chain is the RetryScenario3. That handles this exception that's why we see OnRetry for 502, 504.

One way to solve this problem:

Remove the Or<BrokenCircuitException>() clauses from your retry policies
Define a 4th policy which handles only the BrokenCircuitException

By doing that all four retry policies become mutually exclusive. In other words, their registration order inside the policy chain does not matter because they trigger under different circumstances.

Answer 13 · 2024-07-09T09:35:51.000Z

Thank you so much for your inputs. I commented out //.Or<BrokenCircuitException>()
from all my retry policies and defined a new policy to handle the BrokenCircuitException as below

public AsyncRetryPolicy NoWaitAndRetryAsync()
{
AsyncRetryPolicy noWaitAndRetryPolicy = Policy
.Handle<BrokenCircuitException>()
.RetryAsync(0);
return noWaitAndRetryPolicy;
}

var allRetries = Policy.WrapAsync(RetryScenario1(), RetryScenario2(), RetryScenario3(), waitAndRetryPolicies.NoWaitAndRetryAsync().AsAsyncPolicy(), CircuitBreaker());

I see an exception during the first attempt itself. Not sure what's wrong here. Please help.

14:50:56.239 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
14:50:56.240 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
14:50:56.242 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 1 seconds
14:50:56.242 OnRetryWaitAndRetrySet1 Retry# 1 will be executed after 1 seconds
fail: ClientAPI.TestHarness.Controllers[0]
[Exception] The circuit is now open and is not allowing calls.
fail: ClientAPI.TestHarness.Controllers[0]
[Exception] The circuit is now open and is not allowing calls.
fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
An unhandled exception has occurred while executing the request.
Polly.CircuitBreaker.BrokenCircuitException1[System.Net.Http.HttpResponseMessage]: The circuit is now open and is not allowing calls. at Polly.CircuitBreaker.CircuitStateController1.OnActionPreExecute()

Answer 14 · 2024-07-09T11:15:14.000Z

Whenever the CB is open then you don't have to retry immediately because it will fail with a BrokenCircuitException.

I would recommend the following approach instead:

static IAsyncPolicy<HttpResponseMessage> RetryBrokenCircuit() 
	=> Policy<HttpResponseMessage>
	.Handle<BrokenCircuitException>()
	.WaitAndRetryForeverAsync(sleepDurationProvider: _ => TimeSpan.FromSeconds(5), 
			          onRetry: (ex, ts) => Console.WriteLine("OnRetry for BCE"));

I would wait half as much time as the CB's break duration. (in the example: 10 sec / 2 = 5 sec)
I would not limit the number of retries for the BCE (WaitAndRetryForever)
- Depending on your requirements it may or may not make sense

If I let the application running until it exhausts all the retry attempts for the RetryScenario1 then the output:

OnRetry for 401
OnBreak
OnRetry for 401
OnRetry for BCE
OnRetry for BCE
OnRetry for BCE
OnHalfOpen
OnBreak
OnRetry for 401
OnRetry for BCE
OnRetry for BCE
OnRetry for BCE
OnHalfOpen
OnBreak

Answer 15 · 2024-07-09T18:10:39.000Z

Hi @peter-csala thank you for you inputs. I am trying your suggestions. Currently I am testing all these policies with 2 simulaneous http requests for the same http client. I might face challenges, I will let you know after I succeed with this test. Once again thank you for your support.