Scrape.md is a Python package that allows you to scrape text from any website and generate a comprehensive summary of the text using OpenAI's API. This is then converted into a markdown file that can be used for any purpose.
-
Python 3.6+
-
OpenAI API Key set as an environment variable(OPENAI_API_KEY)
- This can be done with the following command:
export OPENAI_API_KEY="your-key-here"
- You can also add this to your .bashrc or .zshrc file to make it permanent.
You have a few choices for installation but they both start with cloning the repo:
git clone https://github.com/bobbyhiddn/Scrape.md.git
Then you can either install the package using pip:
pip install .
Or you can install the package using the setup.py file:
python setup.py install
To use the package, run the following command:
scrape_md https://www.example.com
This will create a markdown file in the current directory with a filename based on the website you are scraping. For example, if you were scraping https://www.example.com, the file would be named example.md or something like that.