biglocalnews/court-scraper

WI refactor and housecleaning

zstumgoren opened this issue · 0 comments

Related to #64

Needed to update original evictions story

Original source code: https://github.com/Howard-Center-Investigations/evictions/tree/master/scrapers_data/wi_milwaukee

Refactor (or delete?) Wiscourts Site module and class to mirror OdysseySite interface:

  • Remove unused imports
  • Remove the leading _ on import of base._selenium_site_search_mixin
  • Rename class from WiscourtSite -> WiscourtsSite to match the module name
  • Create a wicourts.captcha.CaptchaHelpers class and inherit in pages.search.SearchPage
  • Refactor base selenium code to create base.selenium_helpers.SeleniumHelpers and inherit in pages.search.SearchPage
  • Update refactored WicourtsSite class:
    • Accept placed_id in __init__
    • Set self.url to https://wcca.wicourts.gov/advanced.html in __init__
  • Create WicourtsSite.search method that:
    • Performs a search by date when dates provided (and case numbers are not provided). Test single day and multiple days)
    • Configures selenium (e.g. headless and download_dir)
    • Returns CaseInfo instances containing JSON from caseDetails endpoint
    • Performs case number search when case_numbers provided (this meets minimal CLI requirements)
  • Create wicourts.runner.Runner class and integrate with cli.py:
    • Writes case detail JSON to cache directory
  • Add basic test coverage in a platforms/test_wiscourts_site.py
  • Change wiscourts to wiscourts_site in sites_meta.csv
  • Create WicourtsSite.search_by_date method that:
    • Accepts start/end dates (test single day and multiple days)
    • Uses only requests when case_details=False
    • Uses selenium + requests when case_details=True
    • Basic test coverage