OBOFoundry/purl.obolibrary.org

Document proper log analysis

kltm opened this issue · 1 comments

kltm commented

Now that there are rotating apache logs in an S3 bucket, how does the average operator get at them and use them?

  • Have proper AWS permissions, including API keys
  • Get AWS client tools
  • Get logs in obo-service-logs; currently obo/purl/355fea0db941/2022/03/ would be most useful for what we want
    For example: aws s3 sync s3://DESIRED_LOGS . --include "*" --exclude "*error*"
  • Decompress pigz -d purl.obolibrary.org.access.log.202203*.gz && cat purl.obolibrary.org.access.log.202203* > all-log.txt
  • Use your favorite apache2 log tool; for example goaccess all-log.txt --log-format=COMBINED
kltm commented

Tagging @cmungall