Welcome! This repository contains all the assets for Holistic Evaluation of Language Models, which includes the following features:
- Collection of datasets in a standard format (e.g., NaturalQuestions)
- Collection of models accessible via a unified API (e.g., GPT-3, MT-NLG, OPT, BLOOM)
- Collection of metrics beyond accuracy (efficiency, bias, toxicity, etc.)
- Collection of perturbations for evaluating robustness and fairness (e.g., typos, dialect)
- Modular framework for constructing prompts from datasets
- Proxy server for managing accounts and providing unified interface to access models
To read more:
- Setup: how to run the code
- Code: how to contribute new scenarios or models
- Running the proxy server
- Running the benchmark
- Deployment: for CRFM maintainers of the proxy server