A framework for efficient, consistent and maintainable object-oriented webscraping. Wraps and combines both Selenium Webdriver and BeautifulSoup.
- Websites are best represented using an Object-Oriented Programming approach.
- All websites are made out of pages.
- All pages can be represented as Page objects
- The base of all Page objects can ultimately derive from inheriting a general Page object.
- All webpages are made out of elements.
- All elements can be represented as Element objects
- The base of all Element objects can ultimately derive from inheriting a general Element object.
- General Page and Element objects.
- Selenium-based Page and Element objects with methods that wrap Selenium Webdriver.
- Soup-based Page and Element objects with methods that wrap BeautifulSoup.
Selene is a framework. It provides base Classes that can be used to create website-specific Page
and Element
objects by sub-classing the base Classes. Please refer to the documentation for an example of how this is done.
To install from scratch
- Install Chrome and chromedriver:
sudo apt-get update
apt install -y chromium-chromedriver chromium-browser
- Install:
pip install .