Web Crawler using Spring Boot and Completable Futures & EhCache
An example application using Spring boot and Completable Future & EhCache
Technology stack
- Spring Boot
- Swagger
- Spring Boot Test/JUnit/Mockito/RestAssured
Build Instruction
- Maven
- JDK 8 or above
Run
Internet Without proxy
mvn spring-boot:run
Internet With proxy
| JVM Property | Description |
|---|---|
| http.proxyHost | proxy host name |
| https.proxyHost | https proxy host name |
| http.proxyPort | proxy port number |
| https.proxyPort | https proxy port number |
| http.proxyUser | proxy user name |
| https.proxyUser | https proxy user name |
| http.proxyPassword | proxy user password |
| https.proxyPassword | https proxy user password |
Pass all applicable jvm arguments
mvn -D[jvmProperty]=[value] spring-boot:run
Testing
Endpoint:
http://localhost:8080/crawl?depth=5&breadth=10&url=https://example.com
| Query Param name | Description |
|---|---|
| url | [Mandatory] url to crawl |
| depth | Depth of the page links to be crawled, default is 2 max 10 |
| breadth | Max number of page links to be considered to crawl, default is 10 and max 20 |