Cypress-based tool for crawling real estate offers from the following services:
- Gumtree.pl
- OLX.pl
- Gratka.pl
You need a Node.js environment and ability to run headless Chromium (see requirements for running Cypress)
In repository root run
yarn
If there's a intermittent failure in Cypress installation, try
npx cypress install --force
- Specify parameters in
cypress.env.json
- Run
yarn cypress:open
oryarn cypress:run
- Choose service of your choice (or run all at once - TODO)
- See data/*.json files for results
docker-compose up
- Parametrized tests (
cypress.env.json
config file) - Task for merging all results from
data/*.json
- Add timestamps to every crawled entry
- Task for saving merged results to database (extend config/runtime.json with db connection data)
- Add filtering (exclude/include keywords)
- Add searching with keywords
- Add tests