drupalio/trulens

Evaluation and Tracking for LLM Experiments

🦑 Welcome to TruLens!

TruLens provides a set of tools for developing and monitoring neural nets, including large language models. This includes both tools for evaluation of LLMs and LLM-based applications with TruLens-Eval and deep learning explainability with TruLens-Explain. TruLens-Eval and TruLens-Explain are housed in separate packages and can be used independently.

The best way to support TruLens is to give us a ⭐ on GitHub and join our slack community!

TruLens-Eval

Don't just vibe-check your llm app! Systematically evaluate and track your LLM experiments with TruLens. As you develop your app including prompts, models, retreivers, knowledge sources and more, TruLens-Eval is the tool you need to understand its performance.

Fine-grained, stack-agnostic instrumentation and comprehensive evaluations help you to identify failure modes & systematically iterate to improve your application.

Read more about the core concepts behind TruLens including Feedback Functions, The RAG Triad, and Honest, Harmless and Helpful Evals.

TruLens in the development workflow

Build your first prototype then connect instrumentation and logging with TruLens. Decide what feedbacks you need, and specify them with TruLens to run alongside your app. Then iterate and compare versions of your app in an easy-to-use user interface 👇

Installation and Setup

Install the trulens-eval pip package from PyPI.

pip install trulens-eval

Installing from Github

To install the latest version from this repository, you can use pip in the following manner:

pip uninstall trulens_eval -y # to remove existing PyPI version
pip install git+https://github.com/truera/trulens#subdirectory=trulens_eval

To install a version from a branch BRANCH, instead use this:

pip uninstall trulens_eval -y # to remove existing PyPI version
pip install git+https://github.com/truera/trulens@BRANCH#subdirectory=trulens_eval

Quick Usage

Walk through how to instrument and evaluate a RAG built from scratch with TruLens.

💡 Contributing

Interested in contributing? See our contributing guide for more details.

TruLens-Explain

TruLens-Explain is a cross-framework library for deep learning explainability. It provides a uniform abstraction over a number of different frameworks. It provides a uniform abstraction layer over TensorFlow, Pytorch, and Keras and allows input and internal explanations.

Installation and Setup

These installation instructions assume that you have conda installed and added to your path.

0. Create a virtual environment (or modify an existing one).

conda create -n "<my_name>" python=3  # Skip if using existing environment.
conda activate <my_name>

1. Install dependencies.

conda install tensorflow-gpu=1  # Or whatever backend you're using.
conda install keras             # Or whatever backend you're using.
conda install matplotlib        # For visualizations.

2. [Pip installation] Install the trulens pip package from PyPI.

pip install trulens

Installing from Github

To install the latest version from this repository, you can use pip in the following manner:

pip uninstall trulens -y # to remove existing PyPI version
pip install git+https://github.com/truera/trulens#subdirectory=trulens_explain

To install a version from a branch BRANCH, instead use this:

pip uninstall trulens -y # to remove existing PyPI version
pip install git+https://github.com/truera/trulens@BRANCH#subdirectory=trulens_explain

Quick Usage

To quickly play around with the TruLens library, check out the following Colab notebooks:

• PyTorch:
• TensorFlow 2 / Keras:

For more information, see TruLens-Explain Documentation.