Web scraping group project as a part of Data&AI training at BeCode
Our program will scrape a real estate website (Immoweb) for data about houses and apartments in Belgium. Once the information is fetched it will be cleaned and stored in a CSV file.
- Clone the repo
- Install the required libraries
- Install Selenium WebDriver
- Run main.py
pip3 install -r requirements.txt
Please, follow carefully the installation manual for Selenium WebDriver.
After starting, our program will ask you to select a web driver for Selenium, based on your system. You also have to specify how many search pages you want to scrape thus limiting the number of real estate you need to have information about.
Then a Selenium web driver would grab a list of links via the Immoweb search page starting from the most newer real estate. After collecting individual links (about 30 per page), our program will make an http request to each fetching the data with BeautifulSoup.
The flow of our program is designed in a such way that it will not crash or interrupt if the link or information on it would be incomplete or corrupted in any way. Fetching information is done with multi-threading to be on par with Selenium scraping. We also use a random timeout to not be noticed or banned from the site.
Data would be outputted in 2 files: raw_data.csv and clean_data.csv.
Our program will give these fields about each property: locality, type of property (house or apartment, bungalow, chalet, mansion...), price, type of sale (exclusion or life sale), number of rooms, area, kitchen type, garden, terrace, and swimming pool availability as well as some additional properties.
Values would be stored as strings and numbers (including True and False indicating 1 and 0 correspondingly). Missing or impropriate values would be converting to None in a data frame resulting in am an empty cell in a CSV output file.
Contributions are what make the open-source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature) - Commit your Changes (
git commit -m 'Add some AmazingFeature') - Push to the Branch (
git push origin feature/AmazingFeature) - Open a Pull Request
Distributed under the MIT License.
Please, contact any of the authors via GitHub.
Thank you, Bouman! :)