Datenschule/jedeschule-scraper

Prepare Launch

k-nut opened this issue · 6 comments

k-nut commented

We have done pretty great work on these scrapers lately. There are a couple of things that we still need to do before we can go live. Let's track them here so that we can keep the overview.

  • Store geolocations where possible without geocoding (see #112)
  • Create a simple demo application to explore the data (@knutator2)
  • Offer CSV dump as alternative to using the API
  • Write a blogpost for codefor.de announcing that the API is back
    • Here we can also point out that we are looking for help in e.g. using the new data at jedeschule.de
    • Ensue that GitHub issues are up to date and tagged with good first issue where appropriate
  • Create cronjob on docker host to run scrapers weekly
k-nut commented

A csv dump can be created quite easily using this command:

psql postgres://jedeschule:jedeschule@localhost -c "\copy schools TO '/home/jedeschule/jedeschule-data.csv' DELIMITER ',' CSV HEADER;"
cyroxx commented

CSV downloads are available at https://jedeschule.codefor.de/csv-data/

tursics commented

Ping @knutator2 for "create a simple demo application to explore the data". Do you need help?

k-nut commented

@knutator2 is there anything that we can/should link for your simple demo app? I'd love to close this issue.

knutator2 commented

i think closing this issue is okay. The demo runs under: https://knutator2.github.io/jedeschule-explore/#/dashboard
We can discuss if we continue work on this but from my perspective it is just a good tool to get a feeling if the scraped data looks okay

k-nut commented

i think closing this issue is okay. The demo runs under: https://knutator2.github.io/jedeschule-explore/#/dashboard
We can discuss if we continue work on this but from my perspective it is just a good tool to get a feeling if the scraped data looks okay

Nice, I added a link to the README in PR #116.

I'll hence close this issue.