pichlermeister/immosearch

crawl and analyze search results with pyspider, elasticsearch and kibana

willhaben real estate analysis

1. extract information (pysider)
2. store results (elasticsearch)
3. analyse (kibana)

How to run

1. start docker containers

   cd willhaben
   docker-compse up
   

2. create and run pyspider project

• go to http://localhost:5000
• click "create" and paste content of pyspider_scripts/willhaben_hauser_kaufen_ooe_uu.py
• on overview page: change status to "RUNNING" and click "Run"

3. check kibana

• go to http://localhost:5601
• import saved setting from docs/assets/kibana_export.json
• check "Discover" or "Visualizations" or "Dashboard"

Stack

docker

docker is used as reproduceable easy-to-use runtime environment.

everything can be started via docker-compose up

pyspider

crawls website and extracts information

• pyspider documentation
• supported databases for usage of undocumented databases search in GitHub e.g. elasticsearch+resultdb

elasticsearch

to store results in a kibana-friendly way

• elasticsearch website

kibana

to browse and analyse results

• kibana website

import configuration via docs/kibana_export.json