Zillow Scraping with Python

ATTENTION:

As of 2019, this code no longer works for most users. Zillow is now able to detect the use of all/most automated webdrivers, and will display an unlimited number of CAPTCHA's when the site is launched in a webdriver. I have no interest in putting more work into this project, but am leaving it up to serve as an example of how to webscrape using Selenium with Python.

WARNING: Use this code at your own risk, scraping is against Zillow's TOC

Basic tool for scraping current home listings from Zillow, written in Python using Selenium. The code takes as input search terms that would normally be entered on the Zillow home page. It creates 11 variables on each home listing from the data, saves them to a dataframe, and then writes the df to a CSV file that gets saved to your working directory. Using zip codes as search terms seems to yield the best results, the scraper works at a rate of about 75 zip codes per hour (compared to the Zillow API limit of 1000 homes per 24h).

There are two files, zillow_runfile.py and zillow_functions.py. Clone this repo to your working directory, open the runfile and step through the code line-by-line. The zillow functions are sourced at the top of the runfile.

This tool uses a for loop to iterate over a list of input search terms, scrape the listings of each, and append the results to a dataframe. Function zipcodes_list() allows the user to compile a large list of zip codes to use as search terms, using the package zipcode. For example, st = zipcodes_list(['10', '11', '770']) will yield every US zip code that begins with '10', '11', or '770' as a single list. Object st could then be passed to the scraper.

Some things to keep in mind

You will need to edit the input parameter of function init_driver within zillow_runfile.py to point to the local path of your web driver program (required by Selenium).
The max return for each search term (i.e. each zip code) is 520 home listings.
Zillow will periodically throw up a CAPTCHA page. The script is designed to pause scraping indefinitely until the user has manually completed the CAPTCHA requirements (at which point it should resume scraping).
There tends to be a small amount of NA's on every search, however foreclosure properties seem to be more likely to return NA's. So the more foreclosures there are in a search, the more NA's there will be.

Software Requirements/Info

This code was written using Python 3.5.
Selenium (this can be PIP installed, written using v3.0.2).
The Selenium package requires a webdriver program. This code was written using Chromedriver v2.25.

Example of the output dataframe

df.head(n = 6)