truera/trulens

[FEAT] Adding support of base feedback functions to different languages

sruap1214 opened this issue · 2 comments

sruap1214 commented

Feature Description
Hi. I would like to know what would be the best way to implement the same feedback functions that you have with the prompt in different languages?

Reason
It is needed because after running some test, the score can be very similar if a change the prompt to the other language, but the justification is a mix of different languages.

Importance of Feature
Since this feedback functions some of them depends on language itself, it would nice if there would be a way in which it would easy as just change a parameter to change the language of the prompt, without needing to create a entire feedback function.

dosubot commented

🤖

Based on your request, it seems you want to modify the feedback functions in TruLens Eval v0.24.1 to support multiple languages without creating a new function for each language. This can be achieved by modifying the WithPrompt class or the classes inheriting from it to include a language parameter and dynamically select the prompt template based on the specified language.

Here's a possible approach:

  1. Modify the PromptTemplate class or the way it's used to support multiple languages. This could involve storing templates in different languages and selecting the appropriate one based on a language parameter.

  2. Add a language parameter to the relevant classes, such as WithPrompt or directly in the feedback classes that inherit from it, like Conciseness, Correctness, etc.

  3. Adjust the prompt template selection logic to use the specified language.

Here's an example modification for the WithPrompt class and how it could be applied to a specific feedback class like Conciseness:

class WithPrompt(pydantic.BaseModel):
    prompt: ClassVar[dict]  # Changed to a dictionary to hold prompts in different languages
    default_language: ClassVar[str] = 'en'  # Default language

    @classmethod
    def get_prompt(cls, language=None):
        if language is None:
            language = cls.default_language
        return cls.prompt.get(language, cls.prompt.get(cls.default_language))

class Conciseness(Semantics, WithPrompt):
    prompt: ClassVar[dict] = {
        'en': PromptTemplate.from_template(
            f"""{supported_criteria['conciseness']} Respond only as a number from 0 to 10 where 0 is the least concise and 10 is the most concise."""
        ),
        'fr': PromptTemplate.from_template(
            f"""{supported_criteria['conciseness']} Répondez uniquement par un nombre de 0 à 10 où 0 est le moins concis et 10 est le plus concis."""
        ),
        # Add other languages as needed
    }

# Usage example
language = 'fr'  # This could be dynamically set based on user input or system settings
prompt_template = Conciseness.get_prompt(language)

This approach allows for the dynamic selection of prompt templates based on the language parameter, eliminating the need to create separate feedback functions for each language. You would need to extend this pattern to other feedback classes as necessary and ensure that prompt templates are available in the required languages.

Please note that this is a high-level suggestion and the actual implementation might require additional adjustments depending on the specifics of your codebase and requirements. If you encounter any issues or need further assistance, feel free to ask.

You can find more information about the current implementation of feedback functions in the TruLens Eval v0.24.1 codebase in the trulens_eval/trulens_eval/feedback/init.py file.

Sources

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

  • 👍1
  • 👎1
joshreini1 commented

Hey @sruap1214 - it's probably best here to extend one of our existing providers to a new "custom" feedback function, but then refer to the prompts of the original feedback function as needed.

Here's the docs on extending an existing provider: https://www.trulens.org/trulens_eval/evaluation/feedback_implementations/custom_feedback_functions/#extending-existing-providers