How to Write a Web Scraper in Python
This is the code for the O'Reilly Live Training - Programming with Python: Beyond the Basics presented by Arianne Dee
Before the class, please follow these instructions:
Go to https://www.python.org/downloads/
Click the yellow button at the top to download the latest version of Python.
Follow the prompts and install using the default settings.
The default settings don't add Python to your PATH so your computer doesn't know where to look for it when Python runs (for some inexplicable reason).
Follow the instructions here: Windows Python installer instructions
Follow the instructions here: Add Python to PATH variable in Windows
-
Open the Command Prompt application in Windows or Terminal on Mac or Linux
-
Type
python --version
and press enter -
Type
python3 --version
and press enter -
Type
py --version
and press enter -
One of those commands should print a Python version of 3.6 or higher (whichever version you just downloaded). If it doesn't, you have to follow instructions to add Python to your PATH variable.
Note:
You can now type just the python
, python3
, or py
command
in Command Prompt or Terminal
to run the Python interpreter.
You can also run a .py file by running
python filename.py
An IDE is the program that you write code in. In this class, I will be using PyCharm (Community Edition). I highly recommend it for writing Python code, but you are free to follow along in your IDE of choice.
Download here: https://www.jetbrains.com/pycharm/download/
Install, open, and use the default settings.
If you're viewing this on GitHub already, stay on this page. Otherwise, go to the GitHub repository: https://github.com/ariannedee/python-level-2
Clone the repository.
- Click the "Code" (green) button at the top-right of the page
- Click "Download ZIP"
- Unzip it and move the python-level-2-main folder to a convenient location
Open your IDE and load the course files.
Run the file python-level-2-master/Examples/example_1_review.py
and make sure it runs properly.
If you get an error running that file, follow the instructions here to configure your Python version in PyCharm: Python interpreter setup
Download the PDF of the slides and reference material. These should be in the Resources widget
Yes, but I highly recommend using Python 3. If you are using Python 2, a few commands will be different and you can't use f-strings to format strings. Please see the accompanying resource PDF (page 5) for a list of differences you'll see in this class.
Yes, but it is only recommended if you are already know it and are comfortable navigating to different files and running commands in the command line. If it has syntax highlighting for Python, that is ideal.
Visual instructions here: Python interpreter setup
On a Mac:
- Go to PyCharm > Preferences
On a PC:
- Go to File > Settings
Once in Settings:
- Go to Project: python-level-2 > Project Interpreter
- Look for your Python version in the Project Interpreter dropdown
- If it's not there, click gear icon > Add...
- In the new window, select System Interpreter on the left, and then look for the Python version in the dropdown
- If it's not there, click the ... button and navigate to your Python location
- To find where Python is located, look in these directories
- You may have to search the internet for where Python gets installed by default on your operating system
Yes, email arianne.dee.studios at gmail.com if you have any questions or would like to set up some remote training.