Is Redis required?
Closed this issue · 11 comments
Hi,
I am trying to explore the option of not using Redis with Exceptionless. Is that a possibility or is Redis a requirement?
I think I read somewhere in the documentation that it is optional so I deleted the below environment variables. Is this the wrong approach?
EX_ConnectionStrings__Queue: provider=redis
EX_ConnectionStrings__Redis: server=redis,abortConnect=false
Currently, the Exceptionless logs are saying this:
[16:19:43 ERR] Health check Cache completed after 0.7787ms with status Unhealthy and 'Cache Not Working.'
Thanks for reaching out to us. It's not required but highly recommended as not using it will limit you to run everything in a single process and any non processed queued data will be lost upon app restart. Sounds like there could be some kind of issue with the cache health check when using in memory cache client. Where is this log message coming from?
Thank you very much for the info. The log message was coming from Exceptionless' logs. I figured out the problem. In addition to updating the environment variables, the appsettings.yml needed to be updated as well to remove references to Redis.
The primary reason we are considering not using it is since we self host the applications there is an overhead with maintaining the Linux server that Redis lives on. Using a container would be better in that regard, but that is not a possibility for us currently.
Do you know if running everything in a single process would cause the app to increase the delay between sending events vs when they appear in exceptionless?
That depends on the volume of events that are being sent, but if you were previously only using a single event processor job with Redis then it should be the same or faster.
Thank you! I will close this ticket since my questions were answered.
Are you running into a new issue with Redis?
Just a follow up on the previous comment about the potential for losing queued messages when Redis is not used. Before I review the relevant code, could you validate my assumptions. The one big assumption I am making is that the Exceptionless client stores each event to local disk and then forwards to the Exceptionless web application. If the web server does not respond with success then the client leaves the event on local disk per some retention policy until an eventual call to the web server results in success. Now for the question. When Redis is not enabled, does the web server update Elasticsearch before a success response is sent back to the client or is it using an in-memory queue that could be lost if the web app goes down?
The client by default stores everything in memory unless configured to store it on disk. Depending on the status code returned determines if the event is retried or discarded (Look at the DefaultEventQueue in the clients). The web server streams it to disk immediately, queues an event for processing (with the file path of the saved event to the archive/queue folder) and then returns 202 status code. If you are not using redis you'll have unprocessed content in storage but nothing in the in memory queue so no processing will happen.
Would the evidence of this happening be some old events sitting in the archive/queue folder? Is that only expected to happen if the Exceptionless web application goes down regardless of whether it's a dirty or clean shutdown? I'm just trying to make a judgement call as to whether, for us, the risk of loosing some events is worth the added complexity of the Redis dependency.
Lastly, would you be interested in a pull request that requeues what's on disk if Redis isn't being used?
Yes, having events in the queue folder would mean the in memory queue wasn't processed before restart or you were using redis and the queue item deadlettered. There is an endpoint you can hit to requeue the queue folder (check out the utility controller).
In that case I'm not too worried going without the additional complexity of Redis. We'll enable disk queuing on the client side and requeue the server side queue folder if anything goes wrong. We'll also set up a monitor for the server side queue folder to make sure that nothing hangs around for too long. Does that sound reasonable?
Yes that sounds reasonable, please note adding redis shouldn't add that much overhead as it's super efficient. I'm closing this but let me know if this is still an issue/question and we can reopen if needed.