This benchmark contains code, dataset, and results of classifying Anthropic and OpenAI models over a use case for classifying customer inquiries to customer service in an imaginary financial institution.
First, make sure you have access to Anthropic models via Amazon Bedrock. OpenAI is accessed directly through openAI.
pip install -r requirements.txt
python main.py --action evaluate --llm-names <model_name>
For example:
python main.py --action evaluate --llm-names anthropic.claude-instant-v1
python main.py --action report --llm-names anthropic.claude-instant-v1
See results.csv.
See results per model in detailed-results.
For more read this.
Apache-2.0