Spot interruption event logs incorrect time
aarongorka opened this issue · 3 comments
Describe the bug
When NTH receives a spot interruption notice and logs the interruption time, it logs the current time rather than the actual interruption time.
See also #606
Steps to reproduce
- Set up NTH in Deployment/SQS mode
- Configure EventBridge to send spot termination notices
- Wait for a spot termination notice
- Observe log with "Spot Interruption event received"
Expected outcome
Spot interruption time should be roughly +2 minutes from when the event was received. Ideally we can use the actual interruption time as supplied by AWS rather than guestimating it though.
Application Logs
2023-01-23T00:18:31.086824792Z aws-node-termination-handler-6574867b74-z7l57 {"level":"info","event":{"EventID":"spot-itn-event-62363231313539382d666634352d343631612d646537642d346232353565353832386463","Kind":"SPOT_ITN","Monitor":"SQS_MONITOR","Description":"Spot Interruption event received. Instance i-099447d7bc6f9dde7 will be interrupted at 2023-01-23 00:18:29 +0000 UTC \n","State":"","AutoScalingGroupName":"eks-main-20220713005232834100000009","NodeName":"ip-10-9-139-214.ec2.internal","NodeLabels":null,"Pods":null,"InstanceID":"i-099447d7bc6f9dde7","ProviderID":"aws:///us-east-1b/i-099447d7bc6f9dde7","IsManaged":true,"StartTime":"2023-01-23T00:18:29Z","EndTime":"0001-01-01T00:00:00Z","NodeProcessed":false,"InProgress":false},"time":"2023-01-23T00:18:31Z","message":"Adding new event to the event store"}
2023-01-23T00:20:18.474299920Z aws-node-termination-handler-6574867b74-z7l57 {"level":"info","node_name":"ip-10-9-139-214.ec2.internal","reason":"Spot Interruption event received. Instance i-099447d7bc6f9dde7 will be interrupted at 2023-01-23 00:18:29 +0000 UTC \n","time":"2023-01-23T00:20:18Z","message":"Node successfully cordoned and drained"}
Environment
- NTH App Version: v1.18.2 (Helm chart version 0.18.5)
- NTH Mode (IMDS/Queue processor): Queue processor
- OS/Arch: Amazon Linux 2, x86
- Kubernetes version: 1.24
- Installation method: Helm
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want this issue to never become stale, please ask a maintainer to apply the "stalebot-ignore" label.
This issue was closed because it has become stale with no activity.
This has been released in v1.20.0, chart version 0.22.0