dotnet/orleans

Orleans Message Rejection Exception on Azure: Unable to Connect to Silo Endpoint

SebastianDevelops opened this issue · 0 comments

I’m encountering issues with an Orleans application running on Azure. For some odd reason when the silo instances table contains data, I get the following:

Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.0.0.254:20038:83628028, will retry after 981.0283ms
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99
   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226
   at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230
   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)

Private ports are configured to provide 2 ports and Orleans is configured like this:

var endpointAddress =
    IPAddress.Parse(context.Configuration["WEBSITE_PRIVATE_IP"]!);
var strPorts =
    context.Configuration["WEBSITE_PRIVATE_PORTS"]!.Split(',');
if (strPorts.Length < 2)
    throw new Exception("Insufficient private ports configured.");
var (siloPort, gatewayPort) =
    (int.Parse(strPorts[0]), int.Parse(strPorts[1]));

siloBuilder
    .ConfigureEndpoints(endpointAddress, siloPort, gatewayPort)
    .UseAzureStorageClustering(opt =>
    {
        opt.TableName = orleansSettings.TableStorageConfig.TableName;
        opt.TableServiceClient = new TableServiceClient(settings.TableStorageConfig.ConnectionString);
    })
    .Configure<ClusterOptions>(options =>
    {
        options.ClusterId = settings.ClusterConfiguration.ClusterId;
        options.ServiceId = settings.ClusterConfiguration.ServiceId;
    }).AddAzureBlobGrainStorage(Constants.Orleans.GrainStorageProviderName, (AzureBlobStorageOptions options) =>
  {
      options.BlobServiceClient = new BlobServiceClient(settings.BlobstorageConfig.ConnectionString);
      options.ContainerName = Invariant($"{settings.EnvironmentPrefix}{Constants.Orleans.GrainStorageProviderName}");
  })

Any insights on how I can work past this issue? Besides clearing the instances table?