This project is a web scraper built using PHP 8.2, Laravel 11, Blade templates, and Roach PHP. The web scraper is designed to extract links from specified websites and display it in a user-friendly format.
- Scrapes links from specified websites.
- Displays scraped data using Blade templates.
- Uses Roach PHP for efficient web scraping.
- Built with Laravel 11 for a robust and scalable framework.
- PHP 8.2
- Composer
- Laravel 11
- Node.js and npm (for frontend assets)
- Roach PHP
- PHPUnit
Follow these steps to set up and run the project on your local machine.
git clone https://github.com/thonnysee/scraper_test.git
cd scraper_test
composer install
npm install
npm run build
Copy the example environment file and set your configuration.
cp .env.example .env
Edit the .env file to set your database connection and other configurations.
php artisan key:generate
php artisan migrate
php artisan serve
php artisan queue:work
The application will be available at http://localhost:8000.
There is an instance of docker compose to use a MySQL Database in case it is needed. You will need to have docker running.
Run the next commands:
docker-compose build
docker-compose up
This is the configuration for the default Database:
MYSQL_ROOT_PASSWORD: 'koombea'
MYSQL_DATABASE: 'koombea'
MYSQL_USER: 'koombea'
MYSQL_PASSWORD: 'koombea'
PORT: '3310'
Used port 3310 to avoid conflict with other MySQL instances on the same port
To test the whole application, execute the following command:
php artisan test
Register an user with name, email and password.
After that just log in your account and you can start scraping sites using the input box.
User Dashboard to check all added sites scraped.
Links View
NOTE: The scraper uses the tag "a" for getting all the links so in some cases it will bring only the route not the whole link so be careful when clicking the links. Also be aware it gets all the tags so it could bring some weird stuff
Developed by Antonio Carbajal.
This project is open-sourced software licensed under the MIT license.