truera/trulens

[BUG] Azure OpenAI error with evaluation using TruLlama/Llamaindex

tawalke opened this issue · 14 comments

tawalke commented

Bug Description
What happened?

When using TruLens with Azure OpenAI & LlamaIndex for evaluation we get an error. This code works with OpenAI (non-Azure). The Azure OpenAI endpoint and deployment name has been verified as valid as we can get information back from inferencing using this same Azure OpenAI endpoint. We are using LlamaIndex query engine & TruLens for evaluation.

The Error message we're getting is as follows:

Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.
Groundedness per statement in source:   0%|                                                                                                                                                 | 0/8 [00:00<?, ?it/s]openai request failed <class 'openai.AuthenticationError'>=Error code: 401 - {'statusCode': 401, 'message': 'Unauthorized. Access token is missing, invalid, audience is incorrect (
https://cognitiveservices.azure.com)
, or have expired.'}. Retries remaining=3.
openai request failed <class 'openai.AuthenticationError'>=Error code: 401 - {'statusCode': 401, 'message': 'Unauthorized. Access token is missing, invalid, audience is incorrect (
https://cognitiveservices.azure.com)
, or have expired.'}. Retries remaining=3.
Query: did eddie vedder act?
openai request failed <class 'openai.AuthenticationError'>=Error code: 401 - {'statusCode': 401, 'message': 'Unauthorized. Access token is missing, invalid, audience is incorrect (
https://cognitiveservices.azure.com)
, or have expired.'}. Retries remaining=3.
openai request failed <class 'openai.AuthenticationError'>=Error code: 401 - {'statusCode': 401, 'message': 'Unauthorized. Access token is missing, invalid, audience is incorrect (
https://cognitiveservices.azure.com)
, or have expired.'}. Retries remaining=2.
Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.
openai request failed <class 'openai.AuthenticationError'>=Error code: 401 - {'statusCode': 401, 'message': 'Unauthorized. Access token is missing, invalid, audience is incorrect (
https://cognitiveservices.azure.com)
, or have expired.'}. Retries remaining=2.
openai request failed <class 'openai.AuthenticationError'>=Error code: 401 - {'statusCode': 401, 'message': 'Unauthorized. Access token is missing, invalid, audience is incorrect (
https://cognitiveservices.azure.com)
, or have expired.'}. Retries remaining=2.
Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.

To Reproduce
Which steps should someone take to run into the same error? A small, reproducible code example is useful here.

  • Create an Azure OpenAI deployment
  • Get an Azure OpenAI endpoint and version
  • Create a LlamaIndex query engine object
  • Create an TruLens AzureOpenAI provider
  • Run the following code:
    Upon passing a LlamaIndex query engine object:
provider =AzureOpenAI(
                deployment_name=<Azure OpenAI Deployment Name> 
                endpoint = <Azure OpenAI Endpoint>,
                api_key=<Azure OpenAI API>,
                api_version=<Azure OpenAI API version>
                )

  grounded = Groundedness(groundedness_provider=AzureOpenAI(
          deployment_name=<Azure OpenAI Deployment Name>,
          endpoint = <Azure OpenAI Endpoint>,
          api_key=<Azure OpenAI API>,
          api_version=<Azure OpenAI API version>
          ))

  # Define a groundedness feedback function
  f_groundedness = (
      Feedback(grounded.groundedness_measure_with_cot_reasons)
      .on(context.collect()) # collect context chunks into a list
      .on_output()
      .aggregate(grounded.grounded_statements_aggregator)
  )

  # Question/answer relevance between overall question and answer.
  f_answer_relevance = (
      Feedback(provider.relevance)
      .on_input_output()
  )

  tru_query_engine_recorder = TruLlama(query_engine,
    app_id=run_id,
    feedbacks=[f_groundedness, f_answer_relevance])

  with tru_query_engine_recorder as recording:
      for question in evaluation_questions:
          query_engine.query(question)
        
   rec = recording.get()

Expected behavior
A clear and concise description of what you expected to happen.

Since the OpenAI implementation is the base for Azure OpenAI in TruLens, it is expected that this call would have the same results as the example that happens with OpenAI implementation shown here on this Quickstarts:
https://www.trulens.org/trulens_eval/getting_started/quickstarts/llama_index_quickstart/

Relevant Logs/Tracebacks
Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks. If the issue is related to the TruLens dashboard, please also include a screenshot.

image

Environment:

  • OS: [e.g. MacOS, Windows]
    Tested on both:

    • Windows (Windows 11)
    • Linux (Ubuntu)

  • Python Version

    • 3.11

  • TruLens version
    Versions of other relevant installed libraries

    • trulens-eval==0.27.0
    • llama-index==0.10.11

Additional context
Add any other context about the problem here.

dosubot commented

The error messages you're encountering suggest two distinct issues: an authentication error with Azure OpenAI and a problem with the OpenAICallback class.

For the authentication error, ensure that your Azure OpenAI credentials (api_key and endpoint) are correctly set and valid. These credentials must have the necessary permissions for the services you're accessing. Also, verify that the deployment_name and api_version match exactly with what's configured in your Azure OpenAI service. Incorrect or outdated API versions can lead to authentication failures.

Regarding the callback error, it seems there might be a misconfiguration or incorrect implementation in how the callback functions or classes are defined or used. Ensure that your callback-related code accurately follows the expected patterns and that any custom feedback functions are correctly integrated with the AzureOpenAI provider.

If these steps don't resolve the issues, I recommend checking the Azure OpenAI service's access logs for more detailed error information and ensuring that your Azure subscription and OpenAI service haven't hit any usage limits or restrictions.

Sources

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

  • 👍1
  • 👎2
  • 😕1
tawalke commented

The error messages you're encountering suggest two distinct issues: an authentication error with Azure OpenAI and a problem with the OpenAICallback class.

For the authentication error, ensure that your Azure OpenAI credentials (api_key and endpoint) are correctly set and valid. These credentials must have the necessary permissions for the services you're accessing. Also, verify that the deployment_name and api_version match exactly with what's configured in your Azure OpenAI service. Incorrect or outdated API versions can lead to authentication failures.

Regarding the callback error, it seems there might be a misconfiguration or incorrect implementation in how the callback functions or classes are defined or used. Ensure that your callback-related code accurately follows the expected patterns and that any custom feedback functions are correctly integrated with the AzureOpenAI provider.

If these steps don't resolve the issues, I recommend checking the Azure OpenAI service's access logs for more detailed error information and ensuring that your Azure subscription and OpenAI service haven't hit any usage limits or restrictions.

Sources

About Dosu

@dosubot
We've already done all of the suggested items. Please review error and help us to resolve this as we're unable to use TruLens for evaluation and we need to add this to our pipeline.

  • We have validated there is NO Authentication issue
  • We have validated the code is as recommended
  • We have validated that there is no issue or error on Azure OpenAI logs
  • We can run the internal evaluator we implemented in LlamaIndex with this same Azure OpenAI endpoint
dosubot commented

If you've confirmed that your Azure OpenAI credentials and endpoint configurations are correct and still face authentication errors, it's crucial to ensure that the environment variables for your Azure OpenAI API key and endpoint (AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY) are correctly set and accessible in your application's runtime environment. Additionally, verify that the API key is active and not expired or revoked, and that the endpoint URL matches the one provided in your Azure portal for the OpenAI service.

For the OpenAICallback issue, it seems there might be a misunderstanding or misconfiguration in how callbacks are implemented or expected to work within your application. The OpenAICallback class is designed to handle responses from OpenAI API calls, and it's essential to ensure that any custom callback mechanism you're using is correctly defined and integrated. If OpenAICallback is part of an external library or framework, review the documentation to confirm you're using it as intended and that all necessary configurations for callback mechanisms are properly set up.

Regarding the deployment_name and api_version, these should be correctly provided as part of your AzureOpenAI class instantiation. The deployment_name is a string that represents the name of your deployment and is crucial for making requests to the Azure OpenAI service. The api_version should match the version supported by your Azure OpenAI service, and if it's expected to be provided as an environment variable, ensure it's correctly set.

If after checking these aspects you're still encountering issues, it might be helpful to look into the Azure OpenAI service's access logs for more detailed error information or to verify if there are any usage limits or restrictions that you might have hit.

Sources

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

  • 👍1
  • 👎2
  • 😕1
tawalke commented

If you've confirmed that your Azure OpenAI credentials and endpoint configurations are correct and still face authentication errors, it's crucial to ensure that the environment variables for your Azure OpenAI API key and endpoint (AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY) are correctly set and accessible in your application's runtime environment. Additionally, verify that the API key is active and not expired or revoked, and that the endpoint URL matches the one provided in your Azure portal for the OpenAI service.

For the OpenAICallback issue, it seems there might be a misunderstanding or misconfiguration in how callbacks are implemented or expected to work within your application. The OpenAICallback class is designed to handle responses from OpenAI API calls, and it's essential to ensure that any custom callback mechanism you're using is correctly defined and integrated. If OpenAICallback is part of an external library or framework, review the documentation to confirm you're using it as intended and that all necessary configurations for callback mechanisms are properly set up.

Regarding the deployment_name and api_version, these should be correctly provided as part of your AzureOpenAI class instantiation. The deployment_name is a string that represents the name of your deployment and is crucial for making requests to the Azure OpenAI service. The api_version should match the version supported by your Azure OpenAI service, and if it's expected to be provided as an environment variable, ensure it's correctly set.

If after checking these aspects you're still encountering issues, it might be helpful to look into the Azure OpenAI service's access logs for more detailed error information or to verify if there are any usage limits or restrictions that you might have hit.

Sources

About Dosu

AGAIN We've already done all of the suggested items, we are engineers that work on Azure

Please review error and help us to resolve this as we're unable to use TruLens for evaluation and we need to add this to our pipeline. The error from TruLens is noting an incorrect endpoint where that endpoint is not referencing Azure OAI:

openai request failed <class 'openai.AuthenticationError'>=Error code: 401 - {'statusCode': 401, 'message': 'Unauthorized. Access token is missing, invalid, audience is incorrect (
https://cognitiveservices.azure.com)

  • We have validated there is NO Authentication issue
  • We have validated the code is as recommended
  • We have validated that there is no issue or error on Azure OpenAI logs
  • We can run the internal evaluator we implemented in LlamaIndex with this same Azure OpenAI endpoint and it works correctly.
  • The endpoint that is being noted in TruLens doesn't seem to represent the Azure OAI endpoint
  • There are NO restrictions in our Azure OpenAI endpoint
  • This only happens with our TruLens evaluator code with Azure OpenAI. If the error images is looked at you will find that the Azure OpenAI endpoint provides answers for query before the error.
dosubot commented

Given the detailed information you've provided and the steps you've already taken, it seems the issue might not be with the authentication mechanism itself but possibly with how the Azure OpenAI endpoint is being interpreted or used within the TruLens context. Since you've confirmed that the Azure OpenAI credentials, endpoint configurations, and internal evaluator work correctly outside of TruLens, the focus should shift towards the integration point between TruLens and Azure OpenAI.

Here are some targeted suggestions:

  • Endpoint Configuration in TruLens: Double-check how the Azure OpenAI endpoint is passed into TruLens. Ensure that the endpoint URL is formatted correctly according to TruLens expectations. Sometimes, issues arise not from the credentials but from how the endpoint URL is parsed or utilized within the library.

  • TruLens and Azure OpenAI Integration: Investigate any TruLens-specific configurations or requirements for Azure OpenAI. There might be additional settings or parameters that TruLens expects to establish a successful connection with Azure OpenAI.

  • Custom Callback Mechanism: Since the OpenAICallback error persists, review the implementation of any custom callback mechanisms within your TruLens integration. Ensure that these callbacks are correctly set up to handle responses from Azure OpenAI. It's crucial that the callback mechanism is aligned with both TruLens and Azure OpenAI's expected workflows.

  • Direct Communication with TruLens Support: Given the specificity of the issue and the steps you've already taken, reaching out directly to TruLens support or the community might provide more targeted assistance. They could offer insights into known issues or nuances with Azure OpenAI integration that aren't widely documented.

  • Review Recent Changes: If this integration worked previously or if there have been recent updates to either TruLens or Azure OpenAI, review the change logs for both. A recent update might have introduced changes affecting authentication or endpoint communication.

Your approach and troubleshooting steps so far indicate a deep understanding of the problem space. The issue seems to be at a very specific integration point between TruLens and Azure OpenAI, which might require a more in-depth review of the libraries' internal handling of endpoints and authentication tokens.

Sources

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

  • 👍1
  • 👎2
  • 😕1
tawalke commented

Given the detailed information you've provided and the steps you've already taken, it seems the issue might not be with the authentication mechanism itself but possibly with how the Azure OpenAI endpoint is being interpreted or used within the TruLens context. Since you've confirmed that the Azure OpenAI credentials, endpoint configurations, and internal evaluator work correctly outside of TruLens, the focus should shift towards the integration point between TruLens and Azure OpenAI.

Here are some targeted suggestions:

  • Endpoint Configuration in TruLens: Double-check how the Azure OpenAI endpoint is passed into TruLens. Ensure that the endpoint URL is formatted correctly according to TruLens expectations. Sometimes, issues arise not from the credentials but from how the endpoint URL is parsed or utilized within the library.
  • TruLens and Azure OpenAI Integration: Investigate any TruLens-specific configurations or requirements for Azure OpenAI. There might be additional settings or parameters that TruLens expects to establish a successful connection with Azure OpenAI.
  • Custom Callback Mechanism: Since the OpenAICallback error persists, review the implementation of any custom callback mechanisms within your TruLens integration. Ensure that these callbacks are correctly set up to handle responses from Azure OpenAI. It's crucial that the callback mechanism is aligned with both TruLens and Azure OpenAI's expected workflows.
  • Direct Communication with TruLens Support: Given the specificity of the issue and the steps you've already taken, reaching out directly to TruLens support or the community might provide more targeted assistance. They could offer insights into known issues or nuances with Azure OpenAI integration that aren't widely documented.
  • Review Recent Changes: If this integration worked previously or if there have been recent updates to either TruLens or Azure OpenAI, review the change logs for both. A recent update might have introduced changes affecting authentication or endpoint communication.

Your approach and troubleshooting steps so far indicate a deep understanding of the problem space. The issue seems to be at a very specific integration point between TruLens and Azure OpenAI, which might require a more in-depth review of the libraries' internal handling of endpoints and authentication tokens.

Sources

About Dosu

Please have an engineer to review this as this is a bottleneck and the bot not providing helpful information.
This issue did not exist with previous versions. Please have an engineer validate this issue.
We are engineers that work in Azure and this is not an Azure OAI issue. This endpoint works with other evaluators including Llamaindex native evaluator using the same Azure Open AI endpoint.
There are no restrictions on our Azure subscription.
There is a valid query engine with LlamaIndex
We ran the samples as well with OpenAI and it works with valid OpenAI endpoint but does not work with Azure OAI
We ran the Azure OpenAI endpoint w/other evaluators with no auth errors.

dosubot commented

@joshreini1, this one's a bit out of my wheelhouse, could you step in and provide your guidance?

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

  • 👍1
  • 👎1
piotrm0 commented

Hi @tawalke , I'm having trouble replicating this. Perhaps related is that pip cannot find llama-index==0.10.11 so perhaps there was an issue with it and it got removed from pip. I tried with llama-index==0.10.13 and was able to run your code. Can you include a bit more about the query engine you used?

tawalke commented

Hi @piotrm0:

We can upgrade the llama-index to 0.10.13 to see if this makes a difference. But we pip installed 0.10.11 noted here when we started on the project: https://pypi.org/project/llama-index/0.10.11/ but that was during it's initial release.

We used the Fusion retriever (with basic retriever) and the Flare query engine, see graphic of code here:
image

Here is the code for retriever & query engine

QueryFusionRetriever(
        [inner_retriever],
        llm = QuerySvcEnvs.LLM_MODEL,
        similarity_top_k=QuerySvcEnvs.CONFIGURATION.retriever.top_k,
        num_queries=QuerySvcEnvs.CONFIGURATION.retriever.query_iteration_max, 
        mode="reciprocal_rerank",
        # query_gen_prompt="...",  # we could override the query generation prompt here
        verbose = True,

 query_engine = RetrieverQueryEngine.from_args(retriever, response_mode= ResponseMode.REFINE, llm=QuerySvcEnvs.LLM_MODEL, node_postprocessors=node_postprocessors)

if (QuerySvcEnvs.CONFIGURATION.query_engine.type == QuerySvcEnvs.FLARE
  query_engine = FLAREInstructQueryEngine(query_engine=query_engine, llm=QuerySvcEnvs.LLM_MODEL, max_iterations=QuerySvcEnvs.CONFIGURATION.query_engine.iteration_max, verbose=False)

Please advise what retriever/query engine you used so we can test and additionally if you have any issues using Flare query engine.

piotrm0 commented

Thanks; can you shown the block of code that includes the query engine definition all the way to where you initialize TruLlama with that query engine? I'm unsure from the snippets thus far what was the engine that TruLlama received.

tawalke commented

Thanks; can you shown the block of code that includes the query engine definition all the way to where you initialize TruLlama with that query engine? I'm unsure from the snippets thus far what was the engine that TruLlama received.

@piotrm0 Yes, let me create a file of snippets for you.

joshreini1 commented

@tawalke did you get a chance to create the snippets?

tawalke commented

@joshreini1 Sorry I was out. I'll create the snippets and update the case.

sfc-gh-jreini commented

Closing because stale. @tawalke please reopen if you have continued issues.