This project is a ruby capstone project from the Microverse technical curriculum
Web scraping, also known as web data extraction, is the process of retrieving or “scraping” data from a website. Unlike the mundane, mind-numbing process of manually extracting data, web scraping uses intelligent automation to retrieve hundreds, millions, or even billions of data points from the internet’s seemingly endless frontier. Read more at scrapinghub...
This project demonstrates an example of Web scraping by retrieving data from the Movies | NETFLIX webpage and presenting it to the user.
The Scraper presents retrieved data to the user in 3 phases all of which depend on their selection or response :
-
Phase 1 : A list of movie categories
-
Phase 2 : A list of movies from a particular category
-
Phase 3 : Information about A MOVIE from the list of movies in Phase 2
CATEGORIES MOVIE LIST MOVIE INFORMATION - Ruby
- Nokogiri Gem
- Rspec 3.9
This scraper can be deployed on the following platforms.
B. Online
- You should have Ruby installed on your computer.
- Your computer should be able to run
terminal
orcommand prompt
orbash
orPowershell
commands.
- Click Here to download the zip file of this Scraper.
- Unzip the downloaded file to a folder of your choice.
- Open terminal and navigate to folder containing the unzipped file using
cd
- Run the following command to start the Scraper.
bin/main.rb
- At this point, you should see a welcome message indicating the Scraper has opened.
- Run this command
git clone https://github.com/RNtaate/Web_Scraper.git
to clone the repository on to your local machine. - Run
cd Web_Scraper
to enter into the scraper folder - Run the following command to deploy the scraper
bin/main.rb
- At this point, you should see a welcome message indicating the scraper has opened.
- click the following link Scraper's link
- When the link has been opened, click the
green run button
you will see at the top to run the Scraper.
-
When the scraper is opened, it displays a brief description of how to obtain information and then asks you to enter 'y' or 'yes' if you wish to continue or enter any other key to abort.
-
When you choose to continue, the Scraper will take a couple of seconds to fetch the required information.
-
When it is done retrieving data, it will display a list of movie categories as seen in the CATEGORIES image above.
-
At this point, below the list, the Scraper will ask you to enter a category selection by typing the number appearing before your choice of category.
-
When you enter a valid number, the scraper will display a list of movies contained with in your category of choice as seen in the MOVIE LIST image above
-
Again, at this point, below the movies list, the Scraper will ask you to enter a movie selection by typing the number appearing before your choice of movie.
-
When you make a valid selection, the Scraper will take a couple of seconds to fetch the information about your choice of movie.
-
When it is done retrieving data, it will display the information about the movie you selected in step 6 as seen in the MOVIE INFORMATION image above.
-
After successfully displaying this information, the Scraper will further request you to enter 'y' or 'yes' if you wish to continue or any other key to abort.
-
When you choose to continue, the Scraper will go back to step 3 of this process.
There are two ways of contributing to this project:
-
If you see something wrong or not working, please check the issue tracker section, if that problem you met is not in already opened issues then open the issue by clicking on
new issue
button. -
If you have a solution to that, and you are willing to work on it, follow the below steps to contribute:
- Fork this repository
- Clone it on your local computer by running
git clone https://github.com/RNtaate/Web_Scraper.git
Replace RNtaate with the username you use on github - Open the cloned repository which appears as a folder on your local computer with your favorite code editor
- Create a separate branch off the master branch,
- Write your codes which fix the issue you found
- Commit and push the branch you created
- Raise a pull request, comparing your new created branch with our original master branch here
👤 Roy Ntaate
- Github: @RNtaate
- Twitter: @RNtaate
- Linkedin: roy-ntaate
Give a ⭐️ if you like this project!
- This project was inspired by the Microverse program
This project is MIT licensed.