Project-MONAI/monai-deploy-informatics-gateway

MIG gives null pointer exception and does not retry when MinIO is down

Closed this issue · 2 comments

RemakingEden commented

Description

We currently have an e2e test for resiliency that requires briefly putting MinIO down and seeing how the retry is handled. Right now MIG seems to give an exception and then not retry.

Steps to reproduce

  1. Ensure all required containers are up minus MinIO
  2. Send a dicom through MIG

Expected behavior

MIG would not be able to proceed but would retry the request a configurable number of times with a configurable timeout

Actual behavior

The exception below is given and the payload is deleted instantly

2022-09-29T10:26:55.600071009Z [INFO] ORTHANC -> C-Store response [1]: Success
 2022-09-29T10:26:55.601679439Z [INFO] ORTHANC <- Association release request
 2022-09-29T10:26:55.601900042Z <6> 10:26:55 MONAI[209] => Association=#f1eac5db-c506-4857-bf62-68e9dd8c24b1 10.233.119
 .56:58240 Association release request received.
 2022-09-29T10:26:55.602642291Z [INFO] ORTHANC -> Association release response
 2022-09-29T10:26:55.603977867Z [INFO] Connection closed
 2022-09-29T10:26:58.361016184Z <4> 10:26:58 Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService[4000]
  => File ID=1.3.6.1.4.1.5962.99.1.2968617883.1314880426.1493322302363.3.0/1.3.6.1.4.1.5962.99.1.2968617883.1314880426.
 1493322302363.4.0/1.3.6.1.4.1.5962.99.1.2968617883.1314880426.1493322302363.2.0, Correlation ID=f1eac5db-c506-4857-bf6
 2-68e9dd8c24b1 Failed to upload file 1.3.6.1.4.1.5962.99.1.2968617883.1314880426.1493322302363.3.0/1.3.6.1.4.1.5962.99
 .1.2968617883.1314880426.1493322302363.4.0/1.3.6.1.4.1.5962.99.1.2968617883.1314880426.1493322302363.2.0; added back t
 o queue for retry. System.NullReferenceException: Object reference not set to an instance of an object.    at Minio.Mi
 nioClient.PutObjectAsync(PutObjectArgs args, CancellationToken cancellationToken) in /root/.q/sources/minio-dotnet/Min
 io/ApiEndpoints/ObjectOperations.cs:line 710    at Monai.Deploy.Storage.MinIO.MinIoStorageService.PutObjectUsingClient
 (MinioClient client, String bucketName, String objectName, Stream data, Int64 size, String contentType, Dictionary`2 m
 etadata, CancellationToken cancellationToken)    at Monai.Deploy.Storage.MinIO.MinIoStorageService.PutObjectAsync(Stri
 ng bucketName, String objectName, Stream data, Int64 size, String contentType, Dictionary`2 metadata, CancellationToke
 n cancellationToken)    at Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService.<>c__DisplayClass22_0.
 <<UploadData>b__1>d.MoveNext() in /app/src/InformaticsGateway/Services/Storage/ObjectUploadService.cs:line 226 --- End
  of stack trace from previous location ---    at Polly.AsyncPolicy.<>c__DisplayClass40_0.<<ImplementationAsync>b__0>d.
 MoveNext() --- End of stack trace from previous location ---    at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TR
 esult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPr
 edicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`
 1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)    at Polly.AsyncPolicy.E
 xecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
  at Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService.UploadData(String identifier, StorageObjectMe
 tadata storageObjectMetadata, String source, List`1 workflows, CancellationToken cancellationToken) in /app/src/Inform
 aticsGateway/Services/Storage/ObjectUploadService.cs:line 228    at Monai.Deploy.InformaticsGateway.Services.Storage.O
 bjectUploadService.ProcessObject(FileStorageMetadata blob) in /app/src/InformaticsGateway/Services/Storage/ObjectUploa
 dService.cs:line 179
 2022-09-29T10:27:01.079697046Z <3> 10:27:01 Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler[3014]
  => Correlation ID=f1eac5db-c506-4857-bf62-68e9dd8c24b1 Payload deleted due to upload failure(s) 1.3.6.1.4.1.5962.99.1
 .2968617883.1314880426.1493322302363.3.0.
RemakingEden commented

Tested this and everything seems to be working as expected now

  • 👍1
github-actions commented

🎉 This issue has been resolved in version 0.3.2 🎉

The release is available on: