/async-await-crawler

Web crawler implementation using Python async/await

Primary LanguagePythonMIT LicenseMIT

async-await-crawler

Web crawler implementation using Python 3.5+ async/await

Requirements

  • Python 3.5+
  • pip install -r requirements.txt

Usage

$ python crawl.py www.example.com

Options

  • --out/-o : sitemap output file name
  • --verbose/-v: verbosity level

Docker usage

Build the Docker image:

$ docker build -t async-await-crawler:latest .

Run the crawler

$ docker run -it async-await-crawler http://me.syrex.com