Web Scraper

Web Scraper built using Spring Boot, Angular, and MongoDB. This tool scrapes the data from the website stores it in the database and will show the data in table view with filters and sorting options in the frontend.

Technology Used

Backend: Spring Boot, MongoDB, Swagger2, JUnit
Frontend: Angular, Redux, TypeScript, Angular Material

DEMO

Deployed to Heroku Cloud:

https://scraper-events.herokuapp.com

Note: It is running on a free dyno, so the services go to sleep if not in use. For the first time, it may take some time to respond.

FEATURES

Scrapes the data from the website and pushes the data to database concurrently.
Shows the data in the frontend in table view.
Provides option to sort and filter the data.

Steps to run backend application:

Clone/Download the repository.

Set the environmental variables.

   ACTIVE_PROFILE=<prod/dev>;
   MONGODB_URI=<Connection String>
   CLIENT_URL=<Frontend Client URL>

Build the jar file from springboot-server directory.
```
   ./mvnw install -DskipTests
```

Run the jar file from springboot-server directory.

   java -jar -Djdk.tls.client.protocols=TLSv1.2 target/scraper-data-service.jar

Steps to run frontend application:

Go to file angular-client/src/app/constants/routes.ts and set the backend url.
```
   BACKEND_SERVER_URL=<Backend URL>
```
Run the below command from angular-client directory.
```
   ng serve
```

References

ujjavaldesai07/scraper-springboot-angular-mongodb

Web Scraper