Author: Yingru Li, Jiawei Xu, Zhi-Quan Luo
Welcome to the official implementation of GPT-HyperAgent, a cutting-edge algorithm designed for adaptive foundation models in online decision-making. This repository accompanies our paper Adaptive Foundation Models for Online Decisions: HyperAgent with Fast Incremental Uncertainty Estimation.
GPT-HyperAgent aims to address the challenges of scalable exploration and fast incremental uncertainty estimation in large-scale online decision environments.
- Fast Incremental Uncertainty Estimation: Ensures quick updates and reliable uncertainty quantification with logarithmic per-step computational complexity.
- Scalable Exploration: Efficiently handles large state-action spaces, facilitating robust and adaptive exploration while matching the regret order of exact Thompson sampling.
- Integration with GPT Models: Utilizes the strengths of GPT architectures to enhance decision-making processes in contextual bandits with natural language input.
To get started with HyperAgent, refer to the detailed documentation and examples provided in this repository. For large-scale deep RL benchmarking results and details, visit the szrlee/HyperAgent repository.
We welcome contributions and feedback from the community to help improve and expand the capabilities of HyperAgent.
- To run the experiment on Neural Bandit, use the following command:
python -m scripts.run_hyper --game Neural --method Hyper
- To run the experiment on Online Automated Content Moderation, use this command:
python -m scripts.run_llm --game hatespeech --model-type=linear --llm-name=gpt2
HyperAgent for deep reinforcement learning can be found in the repo szrlee/HyperAgent.
If you find this work useful in your research, please consider citing our paper:
@misc{li2024onlinefoundationagent,
title={Adaptive Foundation Models for Online Decisions: HyperAgent with Fast Incremental Uncertainty Estimation},
author={Yingru Li and Jiawei Xu and Zhi-Quan Luo},
year={2024},
eprint={2407.13195},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2407.13195},
note = {Presentation at ICML 2024 Workshops: (1) "Aligning Reinforcement Learning Experimentalists and Theorists"; (2) "Automated Reinforcement Learning: Exploring Meta-Learning, AutoML, and LLMs"},
}@inproceedings{li2024hyperagent,
title = {{Q-Star Meets Scalable Posterior Sampling: Bridging Theory and Practice via HyperAgent}},
author = {Li, Yingru and Xu, Jiawei and Han, Lei and Luo, Zhi-Quan},
booktitle = {Forty-first International Conference on Machine Learning},
year = {2024},
series = {Proceedings of Machine Learning Research},
eprint = {2402.10228},
archiveprefix = {arXiv},
primaryclass = {cs.LG}，
url = {https://arxiv.org/abs/2402.10228}
}