dotnet/orleans

Graceful shutdown

ArminShoeibi opened this issue · 0 comments

ArminShoeibi commented

Hello there, I am implementing Graceful shutdown, and I want to stop my reminders if I get a SIGTERM from os (right now k8s)

My application will wait for 75 after a SIGTERM to close all the in-flight and in-progress requests.

but 90% of the time I want to deploy my new pods, I will get these sort of exception.

image

{"@timestamp":"2024-05-19T17:47:12.5330983+03:30","level":"Error","messageTemplate":"{Name} failed to stop due to errors at stage {Stage}","message":"\"LocalReminderService\" failed to stop due to errors at stage \"Active (20000)\"","exceptions":[{"Depth":0,"ClassName":"System.Threading.Tasks.TaskCanceledException","Message":"Stopping ReminderService failed because the task was cancelled","Source":"Orleans.Core","StackTraceString":"   at Orleans.Internal.OrleansTaskExtentions.WithCancellation(Task taskToComplete, String message, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 148\n   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStop(CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Lifecycle/SiloLifecycleSubject.cs:line 163","RemoteStackTraceString":null,"RemoteStackIndex":0,"HResult":-2146233029,"HelpURL":null},{"Depth":1,"ClassName":"System.Threading.Tasks.TaskCanceledException","Message":"A task was canceled.","Source":"System.Private.CoreLib","StackTraceString":"   at Orleans.Internal.OrleansTaskExtentions.WithCancellation(Task taskToComplete, String message, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 144","RemoteStackTraceString":null,"RemoteStackIndex":0,"HResult":-2146233029,"HelpURL":null}],"Name":"LocalReminderService","Stage":"Active (20000)","EventId":{"Id":100450},"SourceContext":"Orleans.Runtime.SiloLifecycleSubject","MachineName":"opay-0","ApplicationName":"OPay.API","EnvironmentName":"Development"}
{"@timestamp":"2024-05-19T17:47:12.5431698+03:30","level":"Error","messageTemplate":"Stopping lifecycle encountered an error at stage {Stage}. Continuing to stop.","message":"Stopping lifecycle encountered an error at stage 20000. Continuing to stop.","exceptions":[{"Depth":0,"ClassName":"System.Threading.Tasks.TaskCanceledException","Message":"Stopping ReminderService failed because the task was cancelled","Source":"Orleans.Core","StackTraceString":"   at Orleans.Internal.OrleansTaskExtentions.WithCancellation(Task taskToComplete, String message, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 148\n   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStop(CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Lifecycle/SiloLifecycleSubject.cs:line 163\n   at Orleans.LifecycleSubject.OnStop(CancellationToken cancellationToken) in /_/src/Orleans.Core/Lifecycle/LifecycleSubject.cs:line 194","RemoteStackTraceString":null,"RemoteStackIndex":0,"HResult":-2146233029,"HelpURL":null},{"Depth":1,"ClassName":"System.Threading.Tasks.TaskCanceledException","Message":"A task was canceled.","Source":"System.Private.CoreLib","StackTraceString":"   at Orleans.Internal.OrleansTaskExtentions.WithCancellation(Task taskToComplete, String message, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 144","RemoteStackTraceString":null,"RemoteStackIndex":0,"HResult":-2146233029,"HelpURL":null}],"Stage":20000,"EventId":{"Id":100451},"SourceContext":"Orleans.Runtime.SiloLifecycleSubject","MachineName":"opay-0","ApplicationName":"OPay.API","EnvironmentName":"Development"}
{"@timestamp":"2024-05-19T17:47:12.5643788+03:30","level":"Error","messageTemplate":"{Name} failed to stop due to errors at stage {Stage}","message":"\"ActivationCollector\" failed to stop due to errors at stage \"RuntimeServices (4000)\"","exceptions":[{"Depth":0,"ClassName":"System.Threading.Tasks.TaskCanceledException","Message":"A task was canceled.","Source":"System.Private.CoreLib","StackTraceString":"   at Orleans.Runtime.ActivationCollector.Stop(CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Catalog/ActivationCollector.cs:line 434\n   at Orleans.Runtime.ActivationCollector.<Orleans.ILifecycleParticipant<Orleans.Runtime.ISiloLifecycle>.Participate>b__37_1(CancellationToken cancellation) in /_/src/Orleans.Runtime/Catalog/ActivationCollector.cs:line 444\n   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStop(CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Lifecycle/SiloLifecycleSubject.cs:line 163","RemoteStackTraceString":null,"RemoteStackIndex":0,"HResult":-2146233029,"HelpURL":null}],"Name":"ActivationCollector","Stage":"RuntimeServices (4000)","EventId":{"Id":100450},"SourceContext":"Orleans.Runtime.SiloLifecycleSubject","MachineName":"opay-0","ApplicationName":"OPay.API","EnvironmentName":"Development"}
{"@timestamp":"2024-05-19T17:47:12.5656335+03:30","level":"Error","messageTemplate":"Stopping lifecycle encountered an error at stage {Stage}. Continuing to stop.","message":"Stopping lifecycle encountered an error at stage 4000. Continuing to stop.","exceptions":[{"Depth":0,"ClassName":"System.Threading.Tasks.TaskCanceledException","Message":"A task was canceled.","Source":"System.Private.CoreLib","StackTraceString":"   at Orleans.Runtime.ActivationCollector.Stop(CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Catalog/ActivationCollector.cs:line 434\n   at Orleans.Runtime.ActivationCollector.<Orleans.ILifecycleParticipant<Orleans.Runtime.ISiloLifecycle>.Participate>b__37_1(CancellationToken cancellation) in /_/src/Orleans.Runtime/Catalog/ActivationCollector.cs:line 444\n   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStop(CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Lifecycle/SiloLifecycleSubject.cs:line 163\n   at Orleans.LifecycleSubject.OnStop(CancellationToken cancellationToken) in /_/src/Orleans.Core/Lifecycle/LifecycleSubject.cs:line 194","RemoteStackTraceString":null,"RemoteStackIndex":0,"HResult":-2146233029,"HelpURL":null}],"Stage":4000,"EventId":{"Id":100451},"SourceContext":"Orleans.Runtime.SiloLifecycleSubject","MachineName":"opay-0","ApplicationName":"OPay.API","EnvironmentName":"Development"}

my tasks is in this reminders are not atomic and they are just idempotent things, should I worry about these exceptions?
what should I do In my code to prevent these errors?

How can I prevent reminders from starting after SIGTERM has been sent to my pod? (I mean in that 75-second window.)