/agentspider

The goal of the project is to automate data crawling on web pages using LLM

Primary LanguagePython

AgentSpider

AgentSpider is an automated spider tool, and with the help of LLM, we can make spider simpler

How to use

  1. Install the dependencies

    pip install -r requirements.txt

  2. Change the configs/model_configs.json to set your llm api key

    "api_key": "change-to-your-api-key",

  3. To test the spider, run the following command:

    python test/taskflow_test.py

  4. More, we are developing the automated spider tool, and waiting for it. We need some time to finish it.