/home_scraper

scraper for lerning purposes

Primary LanguagePythonMIT LicenseMIT

home scraper

This package scrapes data from imot.bg

Code style: black

Requirements

Setting up your virtual environment with poetry

From your project directory install one of the two options (for Windows 10 use PowerShell):

  • all dependencies poetry install
  • neglect development dependencies poetry install --no-dev

You should now be able for find your virtual environment in the project directory at the ./venv/ location.

Setting up pre-commit hooks

Run poetry run pre-commit install in the project directory once in order for pre-commit hooks to work automatically at each git commit!

Testing

In the base directory execute poetry run pytest --cov tests/.

Branches

  • master - verified version of the code

TODO

  • export to DB using SQLAlchemy
  • loop over all result pages
  • extract ad data by following each card's link
  • add the following information:
    • agency/broker
    • home type (house)
    • address
    • days_online
    • text
    • floor & area