Based on OpenAI’s LLM API, automatically scrape and analyze data from a specified website, then automatically create and store it in a database table.
This is a practice project, and the author might only industrialize it when the need arises much later.
This project uses the OpenAI LLM API to implement an LLM-based automated web scraper. By specifying the websites to be scraped and the data to be retained in a YAML file using natural language, the scraper will automatically crawl the content described in natural language and store it in a database for subsequent processing.
The implementation includes a reference-counting-based API call load balancer. You can add keys by modifying the YAML file.