Intermittently recives Decrypt failed with OpenSSL error - SSL_ERROR_SSL error

[danleydmello]

Elastic.Clients.Elasticsearch version: 8.17.4

Elasticsearch version: 9.0.2

.NET runtime version: net8.0

Operating system version: windows 2022

Description of the problem including expected versus actual behavior:
Our process uploads lots of documents from the client to our API which makes bulk request to ES.
We see the error intermittently which cause bulk upload to fail and it seems that the requests was waiting for 320s before it failed. The 320s timeout is consistent number for these failures.

System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: The decryption operation failed, see inner exception.
---> Interop+OpenSsl+SslException: Decrypt failed with OpenSSL error - SSL_ERROR_SSL.
---> Interop+Crypto+OpenSslCryptographicException: error:0A000119:SSL routines::decryption failed or bad record mac
  --- End of inner exception stack trace ---
  at Interop.OpenSsl.Decrypt(SafeSslHandle context, Span`1 buffer, SslErrorCode& errorCode)
  at System.Net.Security.SslStreamPal.DecryptMessage(SafeDeleteSslContext securityContext, Span`1 buffer, Int32& offset, Int32& count)
  --- End of inner exception stack trace ---
  at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
  at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
  at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
  --- End of inner exception stack trace ---
  at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
  at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
  at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
  at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
  at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
  at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
  at Elastic.Transport.HttpRequestInvoker.RequestCoreAsync[TResponse](Boolean isAsync, Endpoint endpoint, BoundConfiguration boundConfiguration, PostData postData, CancellationToken cancellationToken)

Not sure whats causing it this failure but it seems its origination from Elastic.Transport

[danleydmello]

Any update on this isse?

[flobernd]

Hi @danleydmello Im on leave for the whole month.

This exception occurs deep in OpenSSL which itself gets called by the .NET HttpClient default implementation. My first guess is that the problem is rather on network side than in the code. Do you use proxies, gateways, load balancers etc in your setup?

[danleydmello]

Hi @flobernd ,

Thanks for replying while on vacation.
Our service is hosted in AKS cluster and it makes direct calls to ElasticSearch cloud APIs using the ElasticSearch .Net client.
Example url is https://0f8847ce33f44bc39f89b3081f7a6d55.uksouth.azure.elastic-cloud.com/6aee702d-2d04-4379-a2a8-28ddbf31adf6/_bulk

So there is no proxy, gateway or any load balancer between ElasticSearch .Net client and ElasticSearch cloud service from our side. I don't know how ElasticSearch cloud search service is hosted.

Regards
Danley

[flobernd]

Closing this since the analysis in the internal SDH indicated that the problem is not caused by the client.