CLI Seo Pocket Crawler

Web Crawler to check few SEO basics.

Use the collected data in your favorite spreadsheet software.

French documentation available : https://piedweb.com/seo/crawler

Install

$ composer create-project piedweb/seo-pocket-crawler

Usage

$ bin/crawler --start="https://piedweb.com"

Other args:

    --start -s
        Define where the crawl start.

    --limit -l
        Define where a depth limit for the crawler (default 5).

    --ignore -i
        Virtual Robots.txt wich will be interpreted for this crawl (could be a
        string or an URL).

    --user-agent -u
        Define the user-agent used during the crawl

    --verbose -v
        Display debugging information (0/1, default 1).

    --wait -w
        In Microseconds, the time to wait between two request. Default : 100000
        (0,1s).

    --cache-method -c
        Keep a copy for each html crawled page : 0 (no),2 (with filename
        corresponding to the ID),1 (with filename corresponding to the Uri).

    --id -i
        Permit to continue or if parameter --restart is set, restart a previous
        crawl. Other args will not be listen.

    --restart -r
        Permit to restart a previous crawl. Values 1 = fresh restart, 2 = restart
        from cache

Testing

$ composer test

Contributing

Please see contributing

Credits

PiedWeb ak Robind4
All Contributors

License

The MIT License (MIT). Please see License File for more information.