if shutdown message is first message in health log then it never expires
rantecki opened this issue · 6 comments
Hi there,
I've been playing with a test setup of Hydra for a few days and doing a lot of starting/stopping of services in the process. I've noticed that Redis is slowly filling up with hydra:service:*:health:log keys from the beginning of time. These keys appear to be persistent.
E.g.
127.0.0.1:6379> keys *:health:log
1) "hydra:service:test-service:fbcef59e18874aa3993b6eff782285ff:health:log"
27.0.0.1:6379> llen hydra:service:test-service:fbcef59e18874aa3993b6eff782285ff:health:log
(integer) 1
27.0.0.1:6379> lindex hydra:service:test-service:fbcef59e18874aa3993b6eff782285ff:health:log 0
"{\"ts\":\"2017-10-24T08:29:27.609Z\",\"serviceName\":\"test-service\",\"type\":\"error\",\"processID\":6672,\"msg\":\"Service is shutting down.\"}"
127.0.0.1:6379> pttl hydra:service:test-service:fbcef59e18874aa3993b6eff782285ff:health:log
(integer) -1
Looking deeper into it, I notice that expire is called on the *:health:log keys as part of updatePresence(). This seems to be run on a timer. Nevertheless, my keys do not expire.
It's looking like I have a situation where the *:health:log key is created (i.e. the first log entry for the service is written) on service shutdown. In this case, the expire command never gets a chance to run on the key before the service shuts down, leaving the key in a persistent state.
I'm not sure if this situation is unique to my test setup, or if I've missed something somewhere. Does anybody else have this issue?
@rantecki Thanks for submitting this! The health:log key is set to expire in one week.
https://github.com/flywheelsports/hydra/blob/master/index.js#L713
Please let us know if that's not working.
@rantecki I pushed hydra and hydra-express 1.4.26 which includes improved key expiration logic. However, that fix won't clean-up old keys. You may need to bring down services and issue a Redis flushdb command. Make sure you're comfortable doing that, and that you first issue a select 15 to switch to the DB that hydra uses.