MIG exports looping
JoeBatt1989 opened this issue · 2 comments
JoeBatt1989 commented
Description
The export request seems to loop quite a lot of times. There was a situation where MIG could not connect to the Destination which resulted in a timeout exception.
Steps to reproduce
- Add AET
- Add destination details (this is a valid destination but where MIG isnt whitelisted to send)
- Add Workflow with an Export Task
- Send data
Expected behavior
Export fails, export.request.complete message is returned with failed status and rabbit messaged acked
Actual behavior
Rabbit message is in unacked state and stack traces are thrown multiple times. Please see below
mocsharp commented
I've configured ORTHANC to reject unknown sources.
After MIG attempted to export 4 times, it acknowledged the request and sent an export complete event.
Please note, similar to #153, if there are N files then IG will attempt to export N x 4 times before acknowledging the request and sending an export complete event.
mdl-ig | info: Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService[525]
mdl-ig | Sending job to ORTHANC@172.29.0.100:4242...
mdl-ig | [DEBUG] [IDLE] --> [CONNECTING]
mdl-ig | [DEBUG] [CONNECTING] --> [REQUESTING ASSOCIATION]
mdl-ig | [DEBUG] Sending association request from MONAISCU to ORTHANC
mdl-ig | [INFO] ORTHANC -> Association request:
mdl-ig | Calling AE Title: MONAISCU
mdl-ig | Called AE Title: ORTHANC
mdl-ig | Remote host:
mdl-ig | Remote port: 0
mdl-ig | Implementation Class: Implementation Class UID [1.3.6.1.4.1.30071.8]
mdl-ig | Implementation Version: fo-dicom 5.0.3
mdl-ig | Maximum PDU Length: 262144
mdl-ig | Async Ops Invoked: 0
mdl-ig | Async Ops Performed: 0
mdl-ig | Presentation Contexts: 2
mdl-ig | Presentation Context: 1 [Proposed]
mdl-ig | Abstract Syntax: Segmentation Storage
mdl-ig | Transfer Syntax: Explicit VR Little Endian
mdl-ig | Presentation Context: 3 [Proposed]
mdl-ig | Abstract Syntax: Segmentation Storage
mdl-ig | Transfer Syntax: Implicit VR Little Endian: Default Transfer Syntax for DICOM
mdl-orthanc | I1006 23:14:58.293983 CommandDispatcher.cpp:333] (dicom) Association Received from AET MONAISCU on IP 172.29.0.50
mdl-orthanc | I1006 23:14:58.294708 main.cpp:318] Incoming connection from AET MONAISCU on IP 172.29.0.50, calling AET ORTHANC
mdl-orthanc | W1006 23:14:58.294748 OrthancConfiguration.cpp:782] Forbidding access from AET "MONAISCU" given its hostname (172.29.0.50) does not match the "DicomModalities" configuration option (informatics-gateway was expected)
mdl-orthanc | W1006 23:14:58.294782 CommandDispatcher.cpp:617] Rejected association for remote AET MONAISCU on IP 172.29.0.50
mdl-ig | [INFO] ORTHANC <- Association reject [result: Permanent; source: ServiceUser; reason: CallingAENotRecognized]
mdl-ig | [INFO] Connection closed
mdl-ig | [DEBUG] Association request from MONAISCU to ORTHANC failed because ORTHANC has rejected it: ORTHANC Permanent ServiceUser
mdl-ig | warn: Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService[522]
mdl-ig | Association rejected.
mdl-ig | [DEBUG] [REQUESTING ASSOCIATION] --> [IDLE]
mdl-ig | fail: Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService[524]
mdl-ig | Error exporting to DICOM destination. Waiting 00:00:00.2500000 before next retry. Retry attempt 1.
mdl-ig | FellowOakDicom.Network.DicomAssociationRejectedException: Association rejected [result: Permanent; source: ServiceUser; reason: CallingAENotRecognized]
mdl-ig | at FellowOakDicom.Network.Client.Advanced.Connection.AdvancedDicomClientConnection.OpenAssociationAsync(AdvancedDicomClientAssociationRequest request, CancellationToken cancellationToken)
mdl-ig | at FellowOakDicom.Network.Client.Advanced.Connection.AdvancedDicomClientConnection.OpenAssociationAsync(AdvancedDicomClientAssociationRequest request, CancellationToken cancellationToken)
mdl-ig | at FellowOakDicom.Network.Client.DicomClient.SendAsync(CancellationToken cancellationToken, DicomClientCancellationMode cancellationMode)
mdl-ig | at FellowOakDicom.Network.Client.DicomClient.SendAsync(CancellationToken cancellationToken, DicomClientCancellationMode cancellationMode)
mdl-ig | at Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService.<>c__DisplayClass14_0.<<HandleDesination>b__1>d.MoveNext() in /app/src/InformaticsGateway/Services/Export/ScuExportService.cs:line 122
mdl-ig | --- End of stack trace from previous location ---
mdl-ig | at Polly.AsyncPolicy.<>c__DisplayClass40_0.<<ImplementationAsync>b__0>d.MoveNext()
mdl-ig | --- End of stack trace from previous location ---
mdl-ig | at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
mdl-ig | [DEBUG] [IDLE] --> [CONNECTING]
mdl-ig | info: Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService[525]
mdl-ig | Sending job to ORTHANC@172.29.0.100:4242...
mdl-ig | [DEBUG] [CONNECTING] --> [REQUESTING ASSOCIATION]
mdl-ig | [DEBUG] Sending association request from MONAISCU to ORTHANC
mdl-ig | [INFO] ORTHANC -> Association request:
mdl-ig | Calling AE Title: MONAISCU
mdl-ig | Called AE Title: ORTHANC
mdl-ig | Remote host:
mdl-ig | Remote port: 0
mdl-ig | Implementation Class: Implementation Class UID [1.3.6.1.4.1.30071.8]
mdl-ig | Implementation Version: fo-dicom 5.0.3
mdl-ig | Maximum PDU Length: 262144
mdl-ig | Async Ops Invoked: 0
mdl-ig | Async Ops Performed: 0
mdl-ig | Presentation Contexts: 2
mdl-ig | Presentation Context: 1 [Proposed]
mdl-ig | Abstract Syntax: Segmentation Storage
mdl-ig | Transfer Syntax: Explicit VR Little Endian
mdl-ig | Presentation Context: 3 [Proposed]
mdl-ig | Abstract Syntax: Segmentation Storage
mdl-ig | Transfer Syntax: Implicit VR Little Endian: Default Transfer Syntax for DICOM
mdl-orthanc | I1006 23:14:58.603092 CommandDispatcher.cpp:333] (dicom) Association Received from AET MONAISCU on IP 172.29.0.50
mdl-orthanc | I1006 23:14:58.603953 main.cpp:318] Incoming connection from AET MONAISCU on IP 172.29.0.50, calling AET ORTHANC
mdl-orthanc | W1006 23:14:58.603999 OrthancConfiguration.cpp:782] Forbidding access from AET "MONAISCU" given its hostname (172.29.0.50) does not match the "DicomModalities" configuration option (informatics-gateway was expected)
mdl-orthanc | W1006 23:14:58.604035 CommandDispatcher.cpp:617] Rejected association for remote AET MONAISCU on IP 172.29.0.50
mdl-ig | [INFO] ORTHANC <- Association reject [result: Permanent; source: ServiceUser; reason: CallingAENotRecognized]
mdl-ig | [INFO] Connection closed
mdl-ig | [DEBUG] Association request from MONAISCU to ORTHANC failed because ORTHANC has rejected it: ORTHANC Permanent ServiceUser
mdl-ig | warn: Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService[522]
mdl-ig | Association rejected.
mdl-ig | [DEBUG] [REQUESTING ASSOCIATION] --> [IDLE]
mdl-ig | fail: Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService[524]
mdl-ig | Error exporting to DICOM destination. Waiting 00:00:00.5000000 before next retry. Retry attempt 2.
mdl-ig | FellowOakDicom.Network.DicomAssociationRejectedException: Association rejected [result: Permanent; source: ServiceUser; reason: CallingAENotRecognized]
mdl-ig | at FellowOakDicom.Network.Client.Advanced.Connection.AdvancedDicomClientConnection.OpenAssociationAsync(AdvancedDicomClientAssociationRequest request, CancellationToken cancellationToken)
mdl-ig | at FellowOakDicom.Network.Client.Advanced.Connection.AdvancedDicomClientConnection.OpenAssociationAsync(AdvancedDicomClientAssociationRequest request, CancellationToken cancellationToken)
mdl-ig | at FellowOakDicom.Network.Client.DicomClient.SendAsync(CancellationToken cancellationToken, DicomClientCancellationMode cancellationMode)
mdl-ig | at FellowOakDicom.Network.Client.DicomClient.SendAsync(CancellationToken cancellationToken, DicomClientCancellationMode cancellationMode)
mdl-ig | at Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService.<>c__DisplayClass14_0.<<HandleDesination>b__1>d.MoveNext() in /app/src/InformaticsGateway/Services/Export/ScuExportService.cs:line 122
mdl-ig | --- End of stack trace from previous location ---
mdl-ig | at Polly.AsyncPolicy.<>c__DisplayClass40_0.<<ImplementationAsync>b__0>d.MoveNext()
mdl-ig | --- End of stack trace from previous location ---
mdl-ig | at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
mdl-ig | [DEBUG] info: Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService[525]
mdl-ig | Sending job to ORTHANC@172.29.0.100:4242...
mdl-ig | [IDLE] --> [CONNECTING]
mdl-ig | [DEBUG] [CONNECTING] --> [REQUESTING ASSOCIATION]
mdl-ig | [DEBUG] Sending association request from MONAISCU to ORTHANC
mdl-ig | [INFO] ORTHANC -> Association request:
mdl-ig | Calling AE Title: MONAISCU
mdl-ig | Called AE Title: ORTHANC
mdl-ig | Remote host:
mdl-ig | Remote port: 0
mdl-ig | Implementation Class: Implementation Class UID [1.3.6.1.4.1.30071.8]
mdl-ig | Implementation Version: fo-dicom 5.0.3
mdl-ig | Maximum PDU Length: 262144
mdl-ig | Async Ops Invoked: 0
mdl-ig | Async Ops Performed: 0
mdl-ig | Presentation Contexts: 2
mdl-ig | Presentation Context: 1 [Proposed]
mdl-ig | Abstract Syntax: Segmentation Storage
mdl-ig | Transfer Syntax: Explicit VR Little Endian
mdl-ig | Presentation Context: 3 [Proposed]
mdl-ig | Abstract Syntax: Segmentation Storage
mdl-ig | Transfer Syntax: Implicit VR Little Endian: Default Transfer Syntax for DICOM
mdl-orthanc | I1006 23:14:59.129541 CommandDispatcher.cpp:333] (dicom) Association Received from AET MONAISCU on IP 172.29.0.50
mdl-orthanc | I1006 23:14:59.130017 main.cpp:318] Incoming connection from AET MONAISCU on IP 172.29.0.50, calling AET ORTHANC
mdl-orthanc | W1006 23:14:59.130048 OrthancConfiguration.cpp:782] Forbidding access from AET "MONAISCU" given its hostname (172.29.0.50) does not match the "DicomModalities" configuration option (informatics-gateway was expected)
mdl-orthanc | W1006 23:14:59.130065 CommandDispatcher.cpp:617] Rejected association for remote AET MONAISCU on IP 172.29.0.50
mdl-ig | [INFO] ORTHANC <- Association reject [result: Permanent; source: ServiceUser; reason: CallingAENotRecognized]
mdl-ig | [INFO] Connection closed
mdl-ig | [DEBUG] Association request from MONAISCU to ORTHANC failed because ORTHANC has rejected it: ORTHANC Permanent ServiceUser
mdl-ig | warn: Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService[522]
mdl-ig | Association rejected.
mdl-ig | [DEBUG] [REQUESTING ASSOCIATION] --> [IDLE]
mdl-ig | fail: Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService[524]
mdl-ig | Error exporting to DICOM destination. Waiting 00:00:01 before next retry. Retry attempt 3.
mdl-ig | FellowOakDicom.Network.DicomAssociationRejectedException: Association rejected [result: Permanent; source: ServiceUser; reason: CallingAENotRecognized]
mdl-ig | at FellowOakDicom.Network.Client.Advanced.Connection.AdvancedDicomClientConnection.OpenAssociationAsync(AdvancedDicomClientAssociationRequest request, CancellationToken cancellationToken)
mdl-ig | at FellowOakDicom.Network.Client.Advanced.Connection.AdvancedDicomClientConnection.OpenAssociationAsync(AdvancedDicomClientAssociationRequest request, CancellationToken cancellationToken)
mdl-ig | at FellowOakDicom.Network.Client.DicomClient.SendAsync(CancellationToken cancellationToken, DicomClientCancellationMode cancellationMode)
mdl-ig | at FellowOakDicom.Network.Client.DicomClient.SendAsync(CancellationToken cancellationToken, DicomClientCancellationMode cancellationMode)
mdl-ig | at Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService.<>c__DisplayClass14_0.<<HandleDesination>b__1>d.MoveNext() in /app/src/InformaticsGateway/Services/Export/ScuExportService.cs:line 122
mdl-ig | --- End of stack trace from previous location ---
mdl-ig | at Polly.AsyncPolicy.<>c__DisplayClass40_0.<<ImplementationAsync>b__0>d.MoveNext()
mdl-ig | --- End of stack trace from previous location ---
mdl-ig | at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
mdl-ig | info: Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService[525]
mdl-ig | Sending job to ORTHANC@172.29.0.100:4242...
mdl-ig | [DEBUG] [IDLE] --> [CONNECTING]
mdl-ig | [DEBUG] [CONNECTING] --> [REQUESTING ASSOCIATION]
mdl-ig | [DEBUG] Sending association request from MONAISCU to ORTHANC
mdl-ig | [INFO] ORTHANC -> Association request:
mdl-ig | Calling AE Title: MONAISCU
mdl-ig | Called AE Title: ORTHANC
mdl-ig | Remote host:
mdl-ig | Remote port: 0
mdl-ig | Implementation Class: Implementation Class UID [1.3.6.1.4.1.30071.8]
mdl-ig | Implementation Version: fo-dicom 5.0.3
mdl-ig | Maximum PDU Length: 262144
mdl-ig | Async Ops Invoked: 0
mdl-ig | Async Ops Performed: 0
mdl-ig | Presentation Contexts: 2
mdl-ig | Presentation Context: 1 [Proposed]
mdl-ig | Abstract Syntax: Segmentation Storage
mdl-ig | Transfer Syntax: Explicit VR Little Endian
mdl-ig | Presentation Context: 3 [Proposed]
mdl-ig | Abstract Syntax: Segmentation Storage
mdl-ig | Transfer Syntax: Implicit VR Little Endian: Default Transfer Syntax for DICOM
mdl-orthanc | I1006 23:15:00.164682 CommandDispatcher.cpp:333] (dicom) Association Received from AET MONAISCU on IP 172.29.0.50
mdl-orthanc | I1006 23:15:00.165427 main.cpp:318] Incoming connection from AET MONAISCU on IP 172.29.0.50, calling AET ORTHANC
mdl-orthanc | W1006 23:15:00.165471 OrthancConfiguration.cpp:782] Forbidding access from AET "MONAISCU" given its hostname (172.29.0.50) does not match the "DicomModalities" configuration option (informatics-gateway was expected)
mdl-orthanc | W1006 23:15:00.165505 CommandDispatcher.cpp:617] Rejected association for remote AET MONAISCU on IP 172.29.0.50
mdl-ig | [INFO] ORTHANC <- Association reject [result: Permanent; source: ServiceUser; reason: CallingAENotRecognized]
mdl-ig | [INFO] Connection closed
mdl-ig | [DEBUG] Association request from MONAISCU to ORTHANC failed because ORTHANC has rejected it: ORTHANC Permanent ServiceUser
mdl-ig | warn: Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService[522]
mdl-ig | Association rejected.
mdl-ig | [DEBUG] [REQUESTING ASSOCIATION] --> [IDLE]
mdl-ig | fail: Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService[530]
mdl-ig | Association rejected with reason CallingAENotRecognized.
mdl-ig | FellowOakDicom.Network.DicomAssociationRejectedException: Association rejected [result: Permanent; source: ServiceUser; reason: CallingAENotRecognized]
mdl-ig | at FellowOakDicom.Network.Client.Advanced.Connection.AdvancedDicomClientConnection.OpenAssociationAsync(AdvancedDicomClientAssociationRequest request, CancellationToken cancellationToken)
mdl-ig | at FellowOakDicom.Network.Client.Advanced.Connection.AdvancedDicomClientConnection.OpenAssociationAsync(AdvancedDicomClientAssociationRequest request, CancellationToken cancellationToken)
mdl-ig | at FellowOakDicom.Network.Client.DicomClient.SendAsync(CancellationToken cancellationToken, DicomClientCancellationMode cancellationMode)
mdl-ig | at FellowOakDicom.Network.Client.DicomClient.SendAsync(CancellationToken cancellationToken, DicomClientCancellationMode cancellationMode)
mdl-ig | at Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService.<>c__DisplayClass14_0.<<HandleDesination>b__1>d.MoveNext() in /app/src/InformaticsGateway/Services/Export/ScuExportService.cs:line 122
mdl-ig | --- End of stack trace from previous location ---
mdl-ig | at Polly.AsyncPolicy.<>c__DisplayClass40_0.<<ImplementationAsync>b__0>d.MoveNext()
mdl-ig | --- End of stack trace from previous location ---
mdl-ig | at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
mdl-ig | at Polly.AsyncPolicy.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
mdl-ig | at Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService.HandleDesination(ExportRequestDataMessage exportRequestData, String destinationName, CancellationToken cancellationToken) in /app/src/InformaticsGateway/Services/Export/ScuExportService.cs:line 95
mdl-ig | info: Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService[505]
mdl-ig | Export task completed with 1 failures out of 1.
mdl-ig | info: Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService[509]
mdl-ig | Sending acknowledgement.
mdl-ig | info: Monai.Deploy.Messaging.RabbitMQ.RabbitMQMessageSubscriberService[10004]
mdl-ig | Sending message acknowledgement for message 6c7ec4a1-5628-4bea-82da-853ed65f363b.
mdl-ig | info: Monai.Deploy.Messaging.RabbitMQ.RabbitMQMessageSubscriberService[10005]
mdl-ig | Ackowledge sent for message 6c7ec4a1-5628-4bea-82da-853ed65f363b.
mdl-ig | info: Monai.Deploy.InformaticsGateway.Services.Export.ScuExportService[511]
mdl-ig | Publishing export complete message.
mdl-ig | info: Monai.Deploy.Messaging.RabbitMQ.RabbitMQMessagePublisherService[10000]
mdl-ig | Publishing message to rabbitmq/monaideploy. Exchange=monaideploy, Routing Key=md.export.complete.
mdl-wm | info: Monai.Deploy.Messaging.RabbitMQ.RabbitMQMessageSubscriberService[10002]
mdl-wm | Message received from queue md.export.complete for md.export.complete.
mdl-wm | info: Monai.Deploy.WorkflowManager.WorkfowExecuter.Services.WorkflowExecuterService[28]
mdl-wm | {"message":"Task Complete","object":{"ExecutionId":"50c358b1-37ae-4dc4-bdb3-fc5a5caaf7cb","TaskId":"export-liver-seg","WorkflowInstanceId":"8a479bcb-0b33-42e0-8fe7-b9a4baca34f2","WorkflowId":"933a967a-593d-4da9-bcb3-040c43d9b6b2","CorrelationId":"5d245cf3-c205-45e3-a739-18f8cc024856","TaskStatus":"Failed","TaskType":"export","TaskStartTime":"2022-10-06T23:14:58.204Z","TaskEndTime":"2022-10-06T23:15:00.1861468Z","TaskStatsObject":{},"PatientDetails":{"patient_id":"00014","patient_name":"LN00014^GN00014","patient_sex":"M","patient_dob":"2006-01-01T00:00:00Z","patient_age":null,"patient_hospital_id":null}}}
mdl-wm | info: Monai.Deploy.Messaging.RabbitMQ.RabbitMQMessageSubscriberService[10004]
mdl-wm | Sending message acknowledgement for message 6c7ec4a1-5628-4bea-82da-853ed65f363b.
mdl-wm | info: Monai.Deploy.Messaging.RabbitMQ.RabbitMQMessageSubscriberService[10005]
mdl-wm | Acknowledge sent for message 6c7ec4a1-5628-4bea-82da-853ed65f363b.
mocsharp commented
Closing as not a bug. Please reopen if MIG doesn't send md.export.complete events.