heraldry-crawler
Script to fill French armorial database, linked with heraldry-search-front project.
Features
- Scrap data from Wikipedia armorials
- Save data in SQLite database
- Index data in Algolia
- Index data in Elasticsearch
Get started
Create .env
file from .env.sample
and complete it with the Algolia app settings in your console.
If you prefer, an Elasticsearch cluster can be used.
NODE_ENV=
ALGOLIA_INDEXING=true # should index data in Algolia
ALGOLIA_APPLICATION_ID= # Aloglia app ID
ALGOLIA_API_KEY= # Algolia API Key
ALGOLIA_INDEX= # Algolia index name
ELASTIC_INDEXING=false # should index data in Elasticsearch
ELASTIC_URL= # ES instance URL
ORIGIN_URL= # heraldry-search-front origin url
To run this project locally, install the dependencies and run the script:
npm install
npm run init-dbs # optional only before first crawl - (re)set database & elastic for emblems
npm run crawl # crawl Wikipedia emblems and index data
npm start # start server to search through ES
Alternatively, you may use Yarn:
yarn
yarn init-dbs # optional only before first crawl - (re)set database & elastic for emblems
yarn crawl # crawl wikipedia emblems and index data
yarn start # start server to search through ES