This tool requires Python3+ and the web search engine API (see link above).
& git clone https://github.com/Amirsorouri00/my-search-engine.git
& cd my-search-engine
& python3.7 -m venv search-engine-venvironment
& source search-engine-venvironment/bin/activate
& pip install -r requirements.txt
$ python manage.py makemigrations
$ python manage.py migrate
And, in another CLI :
sudo docker-compose up
To use the search engine, just type this endpoint in your web browser : http://localhost/
- https://www.freecodecamp.org/news/elasticsearch-with-django-the-easy-way-909375bc16cb/
- https://github.com/adamwattis/elasticsearch-example
- https://docs.scrapy.org/en/latest/topics/benchmarking.html?highlight=redirect%20enabled
- https://en.wikipedia.org/wiki/Sitemaps
- https://docs.scrapy.org/en/latest/topics/extensions.html?highlight=pagecount#extension-settings
- https://docs.scrapy.org/en/latest/topics/practices.html#running-multiple-spiders-in-the-same-process
- https://www.digitalocean.com/community/tutorials/how-to-crawl-a-web-page-with-scrapy-and-python-3
- https://opensourcehacker.com/2011/03/08/installing-and-using-scrapy-web-crawler-to-search-text-on-multiple-sites/
- https://github.com/naqushab/SearchEngineScrapy
- https://github.com/scrapinghub/frontera
- http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-billion-webpages-in-40-hours/
- https://github.com/BruceDone/awesome-crawler
MIT