npm install
nodemon server
and visit http://localhost:8080
nodemon crawler.js
We are using cheerio to scan webpages for links and minisearch to scan the data and give relevant links.
- The crawler first scans the website it is linked with and then saves text linked with the link.
- While doing so the crawler also checks whether link was already saved in the database.
- Since this search engine is especially made for data found on government websites, our crawler filters the links and saves data only related to govenment websites.
- All this data is saved in a json file
- Once we have enough data we can serach throw that data through our website.
Here is a demo for the search engine
