Pair Programming - Ruby Web Scraping
Adam’s learning Ruby! And we want to do some scraping.
https://github.com/reedspool/atp-tour-scraper
Technology
~~Floobits~~ - pair program on different text editors
Zerotier + Tmux - share a terminal (Adam’s logged in to my computer unsafely)
Not using regex for HTML
https://github.com/lorien/awesome-web-scraping/blob/master/ruby.md
Trying Kimurai
Goals
- https://github.com/vifreefly/kimuraframework#installation
- https://www.atptour.com/en/scores/archive/australian-open/580/2019/results
- $ kimurai console –engine selenium_chrome –url https://www.atptour.com/en/scores/archive/doha/580/2019/results
- Write a Ruby file (.rb) that uses Kimurai to open that website and prints the title
- Make a new file in our project
- Bring in (load) kimurai gem
- Use kimurai to open the website
- Get the title and print it
- Loop through all the different “Tournament IDs” and print all their titles
- Make sure we don’t overload their servers
- Instead of print out to terminal, let’s write to a file, so that we don’t have to re-run these processes.
- When kimurai is done parsing all our URLs and writing each into our file, then close our written file
- BUG! why isn’t @file available? (I think I know why, I think they are “class isntance vars”, so @@
What does def (param:) mean? With nothing after colon?
Random stuff
pickledonkeyknife.com https://www.felienne.com/about