/kitas-berlin

Your virtual Jugendämt

Primary LanguageJupyter NotebookMIT LicenseMIT

Die kleine Ziege: Find a Kita in Berlin

Your virtual Jugendämt

Die kleine Ziege is a wrapper that fetch all Kita listings from Kita Navigator Berlin in Python.

Requirements

$ pip3 install numpy
$ pip3 install pandas
$ pip3 install requests
$ pip3 install beautifulsoup4
$ pip3 install selenium
$ pip3 install urllib3
$ brew install geckodriver

Warning

According to your search parameters and location, maybe more than 500 results will appear and it will take a while to fetch all. In my case took more than 2 hours to fetch 672 results.

This happens because the Kitas pages aren't simple HTML vanilla pages, but there's tons of JavaScript code contained, and it means that to scrap the information the page needs to be loaded with all elements and scrolled down.

I'll put some multiprocessing in some moment in the future to speed it up, but for now, enjoy your baby time and wait. Recommended music during the wait: Helge Schneider - Marihuana ist nicht gut

Running Die kleine Ziege in your machine

  1. Download the entire repo in your machine and start a jupyter notebook instance inside of this folder;

  2. Enter in the Kita Navigator website and click in the Kitas Finden. This will open the search (suche) option;

  3. In the suche (search) screen put your address and the radius. After that click in the blue button with will contains the number of kitas available fdor your search;

  4. After that a screen with all the kitas will open. Copy the URL and paste in the base_search_urlpage variable in the first notebook called 01-kita-url-extractor.ipynb;

  5. Check the number of pages of your search and put this number in the number_of_pagination variable;

  6. Run the 01-kita-url-extractor notebook;

  7. After that run the 02-kita-page-scrapper notebook; and

  8. The output will be a Pandas Dataframe.

Why "Die kleine Ziege"?

Apparently, if you have a baby in Berlin everything it's rigged to make your life harder; and find a Kita it's one of those endeavors that the local government uses to test your patience, resilience, and peace of mind. Then, my life partner and I decided to write this wrapper. Die kleine Ziege means "little goat" in German and this is the nickname of our first born.