Bisonai/orakl

Cleanup monitoring DB

Closed this issue · 4 comments

martinkersner commented
  • redis_aggregator_completed 742M rows
  • redis_aggregator_failed 63M rows
  • redis_request_response_completed 30M rows
  • redis_request_response_failed 13K rows
  • redis_vrf_completed 38M rows
  • redis_vrf_failed 15K rows
jay-kim-kor commented

@martinkersner We don't need to keep redis_*_* table even it's related to VRF or RR? If so, I'll remove old data from this table when I clean-up monitor DB.

martinkersner commented

@martinkersner We don't need to keep redis_*_* table even it's related to VRF or RR? If so, I'll remove old data from this table when I clean-up monitor DB.

All of these tables are seem to be persistent storage or Redis, so I am not sure we can delete it through DB directly without affecting Redis itself.

  • Jobs that are completed are not used for anything, not even visualization -> we can clean them up
  • Jobs that are failed (as far as I know) are not going to be retried by BullMQ anymore, BullMQ gave up on such jobs. If that is true, we can clean them all up.

If we have some regular cleaning schedule that would be awesome 💪

  • 👍1
jay-kim-kor commented

TODO

  • If it can remove directly, remove it by SQL cron.
  • If can’t, use redis-cli
    • Test to remove old data
    • Add k8s cronjob
nick-bisonai commented

monitor db will be deprecated after orakl-sentinel implementation