/erowid-scraper

An effective scraper for the erowid website using Python.

Primary LanguagePythonGNU Affero General Public License v3.0AGPL-3.0

Erowid Scraper

1º time trying to create an effective scrapper

For now, its only the basic, I still have to plan the data structure and which info will be scrapped or not, still have a lot to explore.

To-do

  • Get all data from each row of the table
  • Parse all the data from the tables from each page
  • Make requests from each experience page (async and concurrent)
  • Format the data received to Markdown from each page for insertion into a database
  • Data insertions using psycopg2 to PostgreSQL
  • Create a tag for each unique substance found