truera/trulens

[BUG] Using AzureOpenAI provider does not show token usage/price

Dannyjelll opened this issue · 3 comments

Dannyjelll commented

Bug Description
Cost/Token Usage display does not work with AzureOpenai Provider

To Reproduce
Try to evaluate a RAG application using the AzureOpenai provider for any metric like groundedness or answer relevance

Expected behavior
It should display the Token usage/price

Relevant Logs/Tracebacks
I implemented both a custom retriever and one using Trullama. In both cases, I used AzureOpenAI as LLM.:

from trulens_eval.feedback.provider.openai import AzureOpenAI as fAzureOpenai
openai_provider = fAzureOpenai(deployment_name="chat_test")

then
f_qa_relevance = (
Feedback(
openai_provider.relevance_with_cot_reasons,
name = "Answer Relevance",
)
.on(Select.RecordCalls.retrieve.args.query)
.on_output()
)

and so on. The usage is always 0.

Environment:

  • OS: Windows 11 and Archlinux both tested
  • python 3.10.13
  • Trulens 0.28.2

Additional context
none

dosubot commented

The issue with the cost/token usage display not working for the AzureOpenAI provider when evaluating a RAG application for metrics like groundedness or answer relevance is likely due to incomplete or incorrect cost tracking implementation for Azure OpenAI models.

Steps to Resolve:

  1. Verify Environment Variables: Ensure that the following environment variables are correctly set:

    • OPENAI_API_VERSION
    • OPENAI_API_TYPE
    • AZURE_OPENAI_DEPLOYMENT_NAME
    • AZURE_OPENAI_ENDPOINT

  2. Check Cost Tracking Implementation: Review the AzureOpenAI class to ensure that cost tracking is correctly implemented. Verify that the cost tracking logic is integrated and that API responses are parsed to extract cost information.

  3. Debugging: Add logging or debugging statements in the AzureOpenAI class to trace the cost tracking logic and identify any failures.

  4. Update Trulens: Make sure you are using the latest version of Trulens, as updates or bug fixes might address this issue.

If the problem persists, further investigation into the AzureOpenAI class and the _test_llm_provider_endpoint method may be necessary to identify any missing implementations related to cost tracking.

Details

Sources


About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

  • 👍1
  • 👎1
sfc-gh-jreini commented

@sfc-gh-dhuang can you take a look at this?

sfc-gh-dkurokawa commented

This is a duplicate of #1342 so closing in favor of that which I saw first (though came later haha).