php-crawler

This repository has not been updated since a long time and I am not using it anymore. Please feel free to fork it and do whatever you want with it.

A crawler application with a php backend using Laravel, and a js frontend using vuejs, that finds email addresses on the internets.

Given an entry point url, the crawler will search for emails in all the urls available from this entry point domain name. The emails are downloadable as a text file.

Multiple users can start searching for emails without being able see the other users' searches (searches are related to a single user).

Screenshots

Server requirements

PHP >= 7.2.0
OpenSSL PHP Extension
PDO PHP Extension
Mbstring PHP Extension
Tokenizer PHP Extension
XML PHP Extension

Installation

Create a mysql database (default name: crawler)
Install the project with composer:

composer create-project hedii/php-crawler crawler
cd crawler

Open the .env file, check the database credentials, and modify it if needed:

DB_CONNECTION=mysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=crawler
DB_USERNAME=root
DB_PASSWORD=your_password_here

In the .env file, set the application url:

APP_URL=http://crawler.localhost

Build the crawler application

php artisan crawler:build

Point your web server's document / web root to be the public directory: /some/path/crawler/public. The index.php in this directory serves as the front controller for all HTTP requests entering your application. See Laravel documentation. I highly recommend using Laravel Valet if you are using a Mac. Otherwise, check Laravel Homestead.
Done

Usage

Navigate to your php-crawler website
Register a new account
Create a new search
Create more searches
Download the found emails

Testing

composer test

Contributing

All contributions are welcome :)

Please write some tests if you are adding or modifying features.

License

php-crawler is open-sourced software licensed under the MIT license.