/opencompass

OpenCompass is an LLM evaluation platform, supporting evaluation of 20+ models over 50+ datasets, that enables fast, comprehensive benchmarking of large models using efficient distributed evaluation techniques.

Primary LanguagePythonApache License 2.0Apache-2.0



docs license

🌐Website | 📘Documentation | 🛠️Installation | 🤔Reporting Issues

English | 简体中文

Welcome to OpenCompass!

Just like a compass guides us on our journey, OpenCompass will guide you through the complex landscape of evaluating large language models. With its powerful algorithms and intuitive interface, OpenCompass makes it easy to assess the quality and effectiveness of your NLP models.

Introduction

OpenCompass is a one-stop platform for large model evaluation, aiming to provide a fair, open, and reproducible benchmark for large model evaluation. Its main features includes:

  • Comprehensive support for models and datasets: Pre-support for 20+ HuggingFace and API models, a model evaluation scheme of 50+ datasets with about 300,000 questions, comprehensively evaluating the capabilities of the models in five dimensions.

  • Efficient distributed evaluation: One line command to implement task division and distributed evaluation, completing the full evaluation of billion-scale models in just a few hours.

  • Diversified evaluation paradigms: Support for zero-shot, few-shot, and chain-of-thought evaluations, combined with standard or dialogue type prompt templates, to easily stimulate the maximum performance of various models.

  • Modular design with high extensibility: Want to add new models or datasets, customize an advanced task division strategy, or even support a new cluster management system? Everything about OpenCompass can be easily expanded!

  • Experiment management and reporting mechanism: Use config files to fully record each experiment, support real-time reporting of results.

Leaderboard

We provide OpenCompass Leaderbaord for community to rank all public models and API models. If you would like to join the evaluation, please provide the model repository URL or a standard API interface to the email address opencompass@pjlab.org.cn.

image

Dataset Support

Language Knowledge Reasoning Comprehensive Examination Understanding
Word Definition
  • WiC
  • SummEdits
Idiom Learning
  • CHID
Semantic Similarity
  • AFQMC
  • BUSTM
Coreference Resolution
  • CLUEWSC
  • WSC
  • WinoGrande
Translation
  • Flores
Knowledge Question Answering
  • BoolQ
  • CommonSenseQA
  • NaturalQuestion
  • TrivialQA
Multi-language Question Answering
  • TyDi-QA
Textual Entailment
  • CMNLI
  • OCNLI
  • OCNLI_FC
  • AX-b
  • AX-g
  • CB
  • RTE
Commonsense Reasoning
  • StoryCloze
  • StoryCloze-CN (coming soon)
  • COPA
  • ReCoRD
  • HellaSwag
  • PIQA
  • SIQA
Mathematical Reasoning
  • MATH
  • GSM8K
Theorem Application
  • TheoremQA
Code
  • HumanEval
  • MBPP
Comprehensive Reasoning
  • BBH
Junior High, High School, University, Professional Examinations
  • GAOKAO-2023
  • CEval
  • AGIEval
  • MMLU
  • GAOKAO-Bench
  • MMLU-CN (coming soon)
  • ARC
Reading Comprehension
  • C3
  • CMRC
  • DRCD
  • MultiRC
  • RACE
Content Summary
  • CSL
  • LCSTS
  • XSum
Content Analysis
  • EPRSTMT
  • LAMBADA
  • TNEWS

Model Support

Open-source Models API Models
  • InternLM
  • LLaMA
  • Vicuna
  • Alpaca
  • Baichuan
  • WizardLM
  • ChatGLM-6B
  • ChatGLM2-6B
  • MPT
  • Falcon
  • TigerBot
  • MOSS
  • ...
  • OpenAI
  • Claude (coming soon)
  • PaLM (coming soon)
  • ……

Installation

Below are the steps for quick installation. Some third-party features may require additional steps to work properly, for detailed steps please refer to the Installation Guide.

conda create --name opencompass python=3.8 pytorch torchvision -c pytorch -y
conda activate opencompass
git clone https://github.com/InternLM/opencompass opencompass
cd opencompass
pip install -e .
# Download dataset to data/ folder
wget https://github.com/InternLM/opencompass/releases/download/0.1.0/OpenCompassData.zip
unzip OpenCompassData.zip

Evaluation

Please read the Quick Start to learn how to run an evaluation task.

Acknowledgements

Some code in this project is cited and modified from OpenICL.

Citation

@misc{2023opencompass,
    title={OpenCompass: A Universal Evaluation Platform for Foundation Models},
    author={OpenCompass Contributors},
    howpublished = {\url{https://github.com/InternLM/OpenCompass}},
    year={2023}
}