MaximumRetentionDays not work in self-host environment?
gxuan0328 opened this issue · 15 comments
reference to #919, i understand that can setting EX_MaximumRetentionDays environment variable to regular cleanup the expired data.
but it does not work when i trun EX_RunJobsInProcess to false?
is this a correct behavior for the origin design?
i use this(https://github.com/exceptionless/Exceptionless/blob/main/samples/docker-compose.yml) docker-compose.yml to start the exceptionless service.
Yes, I've encountered the same issue where the settings do not take effect after configuration.
@jkhcc11 did you also face the problem that if you use the metion docker-compose.yml to start the exceptionless aplication, then exceptionless/job will throw some error when execution the job because of elasticsearch is not already startup.
Is the job container should wait until elasticsearch is healthy, then start to execute the job? otherwise, all the one time job will failed.
@gxuan0328 I'm sorry, I haven't resolved this issue either, as I am still on the older version. Regrettably, I end up manually deleting it every time because the logs are too large, which is quite troublesome. I am currently testing the latest version in a separate environment to verify this issue and hope to come to a conclusion.
@jkhcc11 currently, i solved this mention cleanup issue through restart job container after all the other componet of exceptionless has successfully running. but i think it is not the best way, i still try to figure out the best practice of it.
@gxuan0328 After restarting the job, when you use Kibana to check the indices, were the logs starting with events-v1- all deleted? If so, I will also try this on the older version. Then, I will use the latest version to figure out the best practice.
@gxuan0328 Okay, I will attempt to try it on the older version.
If you turn run jobs in process off, then you are responsible for running the cleanup jobs periodically. It looks like we should start those jobs if you don't pass any arguments to the job. Do you see logging for it on startup?
When you run in process we run it as a cron job.
services.AddCronJob<CleanupDataJob>("30 */4 * * *");
services.AddCronJob<CleanupOrphanedDataJob>("45 */8 * * *");
services.AddCronJob<DownloadGeoIPDatabaseJob>("0 1 * * *");
services.AddCronJob<MaintainIndexesJob>("10 */2 * * *");
We don't know how everyone will host it. In k8s we use scheduled jobs so that's why we left scheduling out of the out of process jobs.
@niemyjski Hi there, buddy! I couldn't find the corresponding job execution logs on my end. I'm using version 7.x, and Elasticsearch is also 7.x, deployed with Docker Compose. As gxuan0328 mentioned, I tried restarting the job container separately, and it ran normally, and the expired logs were also deleted. I'm not sure if this background task will stop working later on, so I need to keep an eye on it.
We may need an option if you are running only one job container with everything in process to run them as a cron job. Currently they will only be cleaned up on startup which is less than ideal.
@niemyjski am not really sure about RunJobsInProcess this option means?
when should turn it on/off, and what situation suitable for , is there some example about it?
does that means if i turn on the RunJobsInProcess options there is no need to start a independent job container, then jobs will be register in Exceptionless.Web and run CleanupDataJob as cron job. is this correct?
Created a PR that changes the jobs to run on a CRON schedule
@gxuan0328
Yes, if the RunJobsInProcess option is enabled, the job will register a scheduled task when the Web starts. I understand why the job cleared the data after it started yesterday; it's because the standalone job only checks whether it needs to clear old data at startup. They updated a PR yesterday that resolved this issue.
@jkhcc11 @niemyjski @ejsmith Thanks for helping me a lot.
After i try turn on the RunJobsInProcess options, all the job execute expectedly, but i stiil face a problem that if job start with Exceptionless.Web then nothing about the job execution detail wiil be show, is there any setting can show logs in this situation?
The default log level is warning so info logs won't be displayed. This is probably what you are running into. You could follow the troubleshooting docs which describes how to change the log levels.