nicksamuel/screamingfrog-cloud

A basic guide to automating Screaming Frog in the Cloud a Ubntu droplet from Digital Ocean with example code that I used.

Screaming Frog in the Cloud - A Super Easy Way To Automate and Download Your Crawls

Getting Started

These are some fairly basic instructions which should help you get Screaming Frog up and running in the cloud.

I don't claim that this is the best, most scalable or most efficient way of running SF in the Cloud; the real goal of this little guide is to demonstrate how easy it is to deploy SF in the Cloud, and to hopefully encourage other SEOs out there to try it!

Steps are as follows:

1. Set up a Ubuntu VPS - Example uses Digital Ocean.
2. Install Screaming Frog + Dependencies - This can either be done manually or by using Fili Wiese's script
3. Install Apache - to browse your crawls
4. Create Cron Jobs - Configure and Schedule your Crawls
5. Sit back and Relax - Get your files using your web browser

Please note that you don't have to use Digital Ocean or any hosting provider in particular, however it's probably sensible to stick with a Ubuntu VPS to minimise any potential compatability issues with Screaming Frog.

Step 1: Set up a Ubuntu VPS

Step 2: Install Screaming Frog

Step 3: Install Apache

Step 4: Create Cron Jobs

Step 5: Get Your Files

Further Reading

Below are some resources which either inspired me or directly helped me write this guide:

• https://searchengineland.com/how-to-run-screaming-frog-seo-spider-in-the-cloud-in-2019-317416

• https://github.com/fili/screaming-frog-on-google-compute-engine

• https://www.screamingfrog.co.uk/seo-spider/user-guide/general/#commandlineoptions

• https://www.digitalocean.com/community/tutorials