desihub/nightwatch

clean up nightwatch disk usage at KPNO

sybenzvi opened this issue · 1 comments

sybenzvi commented

@daqiii reports that Nightwatch is taking up 11T in the folder /exposures/nightwatch on the DESI cluster. Check that @sbailey's cleanup cronjob is running on desi-7 and is correctly deleting old files. If needed, make the cleaning script more aggressive about deleting intermediate FITS files produced during processing.

sybenzvi commented

Adding the following lines to the script ~datasystems/users/sjbailey/kpno_nightwatch_cleanup.sh removes all preproc HTML files older than 2 weeks and replaces them with an HTML page with a back button. The per-exposure disk usage drops from a minimum of 80MB to 4MB, so this should free a considerable amount of space. I'll close the ticket after the cleanup cronjob runs tomorrow at 9 am MT.

#- Delete preproc HTML files and replace with a simple redirect page
clean_preproc() {
    dir=`dirname $1`
    preproc=${dir}/preproc.html
    if [ ! -f ${preproc} ]; then
        # echo "Creating ${preproc}"
cat << EOF > ${preproc}
<html>
  <head>
    <title>Preproc output: not found.</title>
  </head>
  <body>
    <p>Preproc output removed.</p>
    <button onclick="history.back()">Go Back</button>
  </body>
</html>
EOF
    fi

    rm $1
    ln -s ${preproc} $1
}

export -f clean_preproc
find /exposures/nightwatch/20?????? -name \*preproc-\*.html -mtime +14 -exec bash -c 'clean_preproc "$1"' _ {} \;