/YiVal

πŸš€ Evaluate and Evolve.πŸš€ YiVal is an open source GenAI-Ops framework that allows you to manually or automatically tune and evaluate your AIGC prompts, retrieval configs and fine-tune the model params all at once with your preferred choices of test dataset generation, evaluation algorithms and improvement strategies.

Primary LanguagePythonApache License 2.0Apache-2.0

πŸ§šπŸ»β€οΈ YiVal

Website Β· Producthunt Β· Documentation

⚑ Build any Generative AI application with evaluation and improvement ⚑

πŸ‘‰ Follow us: Twitter | Discord

Downloads License: MIT GitHub star chart Dependency Status Open Issues

πŸ€” What is YiVal?

YiVal is an GenAI-Ops framework that allows you to iteratively tune your Generative AI model metadata, params, prompts and retrieval configs all at once with your preferred choices of test dataset generation, evaluation algorithms and improvement strategies.

Check out our quickstart guide! β†’

πŸ“£ What's Next?

Expected Features in Sep

  • Add ROUGE and BERTScore evaluators
  • Add support to midjourney
  • Add support to LLaMA2-70B, LLaMA2-7B, Falcon-40B,
  • Support LoRA fine-tune to open source models

πŸš€ Features

πŸ”§ Experiment Mode: πŸ€– Agent Mode (Auto-prompting):
Workflow Define your AI/ML application ➑️ Define test dataset ➑️ Evaluate πŸ”„ Improve ➑️ Prompt related artifacts built βœ… Define your AI/ML application ➑️ Auto-prompting ➑️ Prompt related artifacts built βœ…
Features 🌟 Streamlined prompt development process
🌟 Support for multimedia and multimodel
🌟 Support CSV upload and GPT4 generated test data
🌟 Dashboard tracking latency, price and evaluator results
🌟 Human(RLHF) and algorithm based improvers
🌟 Service with detailed web view
🌟 Customizable evaluators and improvers
🌟 Non-code experience of Gen-AI application build
🌟 Witness your Gen-AI application born and improve with just one click
Demos - Animal story with MidJourney 🐯 Open In Colab
- Model Comparison in QA ability 🌟Open In Colab
- Startup Company Headline Generation BotπŸ”₯ Open In Colab
- Automate Prompt Generation with Retrieval MethodsπŸ”₯ Open In Colab

Model Support matrix

We support 100+ LLM ( gpt-4 , gpt-3.5-turbo , llama e.g.).

Different Model sources can be viewed as follow

Model llm-Evaluate Human-Evaluate Variation Generate Custom func
OpenAI βœ… βœ… βœ… βœ…
Azure βœ… βœ… βœ… βœ…
TogetherAI βœ… βœ… βœ… βœ…
Cohere βœ… βœ… βœ… βœ…
Huggingface βœ… βœ… βœ… βœ…
Anthropic βœ… βœ… βœ… βœ…
MidJourney βœ… βœ…

To support different models in custom func(e.g. Model Comparison) , follow our example

To support different models in evaluators and generators , check our config

Installation

pip install yival

Demo

Multi-model Mode

Yival has multimodal capabilities and can handle generated images in AIGC really well.

Find more information in the Animal story demo we provided.

yival run demo/configs/animal_story.yml

pic

Basic Interactive Mode

To get started with a demo for basic interactive mode of YiVal, run the following command:

yival demo --basic_interactive

Once started, navigate to the following address in your web browser:

http://127.0.0.1:8073/interactive

Click to view the screenshot

Screenshot 2023-08-17 at 10 55 31 PM

For more details on this demo, check out the Basic Interactive Mode Demo.

Question Answering with expected result evaluator

yival demo --qa_expected_results

Once started, navigate to the following address in your web browser: http://127.0.0.1:8073/

Click to view the screenshot Screenshot 2023-08-18 at 1 11 44 AM

For more details, check out the Question Answering with expected result evaluator.