truera/trulens

When Custom retriever and query engine with llamaindex,get ERROR

rushmj opened this issue · 2 comments

rushmj commented

Hello
In llamaindex,I use "BaseRetriever" and "CustomQueryEngine" class to custom Retriever with Neo4j query and use it to custom the query engine,then I put the custom_query_engine variable to "context = App.select_context(custom_query_engine)"
I get ERROR:"Could not determine context from unrecognized app type {type(app)}."
it seem it only support the QueryEngine from official llamaindex package.
How can I do to make my customQueryEngine work?
Thank you.

dosubot commented

Hey @rushmj, great to see you diving into another challenge! Hope you're doing well.

To make your custom QueryEngine work with llamaindex when using the BaseRetriever and CustomQueryEngine class, and to resolve the error stating "Could not determine context from unrecognized app type {type(app)}," you need to ensure that your custom QueryEngine is properly instrumented and recognized by the TruCustomApp framework. The error suggests that the TruCustomApp constructor is unable to recognize the type of the app argument passed to it, likely because the app does not meet the expected structure or is not properly instrumented.

Here are the steps to resolve the issue:

  1. Ensure Proper Instrumentation: Make sure that all relevant methods in your custom QueryEngine and any other components it interacts with are decorated with the @instrument decorator. This is crucial for TruCustomApp to recognize and monitor the methods.

  2. Verify Component Accessibility: According to the limitations mentioned in the documentation, all tracked (instrumented) components must be accessible through other tracked components. Ensure that your QueryEngine and any of its sub-components (if any) are accessible in this manner. If your QueryEngine contains non-instrumented classes that have instrumented methods, this could lead to the error you're encountering.

  3. Check for Main Method Specification: If your custom QueryEngine has a main method that acts as the entry point for processing queries, ensure that this method is correctly specified when initializing TruCustomApp. The main_method argument should be a bound method (e.g., app_instance.main_method rather than AppClass.main_method). This is important for TruCustomApp to correctly identify and instrument the main method of your application.

  4. Use methods_to_instrument if Necessary: If there are methods that you cannot decorate directly with @instrument (for example, because they are part of a third-party library), you can use the methods_to_instrument argument of the TruCustomApp constructor to explicitly specify these methods for instrumentation. This argument takes a dictionary where keys are method references and values are the paths to these methods within the app.

  5. Review Error and Warning Messages: Pay close attention to any error or warning messages generated by TruCustomApp during initialization or operation. These messages can provide valuable insights into what might be going wrong. For example, if you see warnings about functions not found during the instrumentation walk, it indicates that TruCustomApp is unable to locate these methods in the structure of your app, which might be causing the error you're seeing.

By following these steps and ensuring that your custom QueryEngine and its components are correctly instrumented and recognized by TruCustomApp, you should be able to resolve the error and make your custom QueryEngine work with llamaindex.

For more details, you can refer to the trulens_eval/trulens_eval/tru_custom_app.py file in the TruLens repository.

Sources

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

  • 👍1
  • 👎1
joshreini1 commented

Hey @rushmj can you select the context more manually with a feedback selector? Not sure select_context helper will work in this case.

If you're creating the new custom classes youself, I'd recommend using our custom app instrumentation with the @instrument decorator. This will give you easy access to the args and return of any method you tag with that decorator for evals. For an example, see the base quickstart: https://www.trulens.org/trulens_eval/getting_started/quickstarts/quickstart/

  • 👍1