nasa/opera-sds-ops

[New Feature]: Solution to clean up SDS files that are accumulating

riverma opened this issue · 3 comments

riverma commented

Checked for duplicates

Yes - I've already checked

Alternatives considered

No - I haven't considered

Related problems

No PCM tickets as of yet, but some PGE CI machine tickets like this one.

Describe the feature request

We currently do not have a procedure / mechanism to allow OPS to be confident the SDS will not run out of disk, have performance issues, or have extreme costs in case certain files in the system are not deleted. We need a solution for this.

Files like:

  • Elasticsearch docs that are out-of-date (reference delivered products)
  • Log files on disk
  • Temporary files outside of /tmp on SDS machines
  • Outdated S3 files
riverma commented

@niarenaw - correct me if I'm wrong, we don't have any cleanup procedures as of yet do we?

riverma commented

Per suggestion from @philipjyoon - we need to include regular cleanups of Elasticsearch as well.

LucaCinquini commented

Nick to check with PCM team to decide what information needs to be cleared.