/AnyTool

Primary LanguagePythonApache License 2.0Apache-2.0

AnyTool

Static Badge

This is the implementation of the paper AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls Figure

πŸ”§ Installation

βœ… Dependencies

Require Python 3.9+

πŸš€ Quick install

pip install -r requirements.txt

πŸ”† Preparation

OPENAI API config and the ToolBench key

Fill your OpenAI GPT-4 API config and toolbench key into the config.py (see config_example.py as an example). We use Azure OpenAI for all our experiments. You can modify it according to your own configuration.

Fill out the form to get the toolbench key. If you want to use your own RapidAPI key, you can put your key in the rapidapi_key_list.json (see rapidapi_key_list_example.json as an example)

ToolBench

Download the ToolBench data using the following link: Google Drive or Tsinghua Cloud. Decompress the data.zip and the file structure is as follows:

β”œβ”€β”€ /data/
β”‚  β”œβ”€β”€ /instruction/
β”‚  β”œβ”€β”€ /answer/
β”‚  β”œβ”€β”€ /toolenv/
β”‚  β”œβ”€β”€ /retrieval/
β”‚  β”œβ”€β”€ /test_instruction/
β”‚  β”œβ”€β”€ /test_query_ids/
β”‚  β”œβ”€β”€ /retrieval_test_query_ids/
β”‚  β”œβ”€β”€ toolllama_G123_dfs_train.json
β”‚  └── toolllama_G123_dfs_eval.json
β”œβ”€β”€ /reproduction_data/
β”‚  β”œβ”€β”€ /chatgpt_cot/
β”‚  β”œβ”€β”€ /chatgpt_dfs/
β”‚  β”œβ”€β”€ ...
β”‚  └── /toolllama_dfs/

For more details, please refer to ToolBench.

Prepare the API data

You should prepare the ToolBench data first. Make sure you have the directory of data/toolenv/tools

export PYTHONPATH=./
python preprocess/extract_api_details.py
python preprocess/extract_category_tool_details.py
python preprocess/extract_tool_database.py

AnyToolBench

Generation script

export PYTHONPATH=./
python scripts/anytoolbench_generation.py --output_path atb_data/anytoolbench_new.json

We provide sample data in anytoolbench.json file.

The data look like

"query": "Can you provide detailed information about \"The Incredible Hulk\" movie that was released in 2008, including its plot, genres, and how it's evaluated by audiences, and also tell me the current timezone for Los Angeles, USA?",
"final_answer": "The Incredible Hulk (2008) is about scientist Bruce Banner who searches for an antidote to his unbridled rage, the Hulk, but faces new foes when forced back to civilization. GENRES: Sci-Fi, Action, Adventure. AUDIENCE SCORE: 6.2/10. The current timezone for Los Angeles, USA, is America/Los_Angeles.",
"query_id": "1000006",
"gt_api_list": [
            {
                "category_name": "Movies",
                "tool_name": "Advanced Movie Search",
                "api_name": "Search by Name"
            },
            {
                "category_name": "Location",
                "tool_name": "Timezone By API-Ninjas",
                "api_name": "/v1/timezone"
            }
        ],

πŸš— Run AnyTool

Experiment on ToolBench, take G1-I as an example.

export PYTHONPATH=./
python scripts/main.py --output_dir result/test_instruction/G1_instruction --query_path data/test_instruction/G1_instruction.json --max_api_number 64

Experiment on AnyToolBench

export PYTHONPATH=./
python scripts/main.py --output_dir result/anytoolbench --query_path anytoolbench.json -max_api_number 64

The pass rate can be found in the success_cnt.txt under the output directory.

πŸ“ Experiment Results

Main results on the filtered ToolBench. We use pass rate defined in Eq 2 and illustrated in Figure 4(b) in our paper, as the metric. All results are reproduced. *: OpenAI’s text-embedding-ada-002; Ref.: reference; Avg.: average; SR: self-reflective.

Model API Retriever Solver Use Ref. APIs G1 I (%) G1 T (%) G1 C (%) G2 I (%) G2 C (%) G3 I (%) Avg. (%)
ToolLLM OpenAI TE* ToolLLaMA w/ DFSDT 8.7 6.8 12.0 4.7 8.2 10.5 8.5
ToolLLM ToolLLM's ToolLLaMA w/ DFSDT 28.4 26.3 38.4 21.5 15.1 7.7 22.9
ToolLLM ToolLLM's GPT-4 w/ DFSDT 42.6 46.2 51.4 23.4 24.5 2.6 31.8
ToolLLM None ToolLLaMA w/ DFSDT βœ“ 29.4 31.8 37.1 19.6 22.4 13.2 25.6
GPT-4 None GPT-4 w/ CoT βœ“ 31.3 34.8 47.1 27.1 34.7 2.6 29.6
GPT-4 None GPT-4 w/ DFSDT βœ“ 36.5 49.2 51.4 38.3 39.8 18.4 38.9
GPT-4 Plain Agent GPT-4 w/ DFSDT 13.9 23.5 17.6 13.9 9.2 13.2 15.2
GPT-4 AutoGen-RAG GPT-4 w/ DFSDT 14.8 19.7 19.7 7.4 9.2 7.9 13.1
GPT-3.5 None GPT-3.5 w/ CoT βœ“ 37.5 37.1 42.9 24.3 22.4 5.3 28.3
GPT-3.5 None GPT-3.5 w/ DFSDT βœ“ 39.1 40.2 48.6 31.8 25.5 15.8 33.5
AnyTool (Ours) SR Agent SR GPT-4 w/ DFSDT 52.2 61.4 67.6 58.9 45.9 63.2 58.2

Results on our AnyToolBench. All models use DFSDT implementation in the solver. SR: self-reflective; PR: pass rate

Method API Retriever Solver PR (%)
ToolLLM ToolLLM’s ToolLLaMA 18.9
ToolLLM ToolLLM’s GPT-4 36.6
GPT-4 Plain Agent GPT-4 14.0
AnyTool (Ours) SR Agent SR GPT-4 73.8

πŸ‘¨β€πŸ« Acknowledgement

This repo is built on ToolBench.

πŸ“‘Citation

If you find this project is helpful for your research, consider citing our paper

@article{du2024anytool,
  title={AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls},
  author={Du, Yu and Wei, Fangyun and Zhang, Hongyang},
  journal={arXiv preprint arXiv:2402.04253},
  year={2024}
}