using the openai API for a BlackBox model for non OPENAI hosted platforms

[psteinb]

Hi,

thanks for providing the community with this library. I believe uncertainties of LLM queries are an important topic. I tried to play around with this library and am a bit stuck. So I'd like to use a remote model that is accessible through the openai library. For this, I have to provide a custom OPENAI_API_BASE and my OPENAI_API_KEY. However, the library tells me that it doesn't know how to query the remote model?

Here is the code that I drafted given your example:

def main():
    print(f":: black box test, using Mistral-7B-Instruct-v0.2 from {os.environ["OPENAI_API_BASE"]}")
    model = BlackboxModel(openai_api_key=os.environ["OPENAI_API_KEY"], model_path="Mistral-7B-Instruct-v0.2", parameters={"openai_api_base": os.environ["OPENAI_API_BASE"]})

    print(model.parameters)

    print(":: using estimator EigValLaplacian")
    estimator = EigValLaplacian(verbose=True)
    answer = estimate_uncertainty(
        model, estimator, input_text="When did Albert Einstein die?"
    )
    print(">>",answer)

So I get the following error:

:: using estimator EigValLaplacian
Traceback (most recent call last):
  File "/home/steinb95/development/lm-polygraph/lm-polygraph/examples/./black_box.py", line 23, in <module>
    main()
  File "/home/steinb95/development/lm-polygraph/lm-polygraph/examples/./black_box.py", line 16, in main
    answer = estimate_uncertainty(
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/steinb95/development/lm-polygraph/lm-polygraph/src/lm_polygraph/utils/manager.py", line 166, in estimate_uncertainty
    man()
  File "/home/steinb95/development/lm-polygraph/lm-polygraph/src/lm_polygraph/utils/manager.py", line 400, in __call__
    batch_stats = self.calculate(batch_stats, self.stat_calculators, inp_texts)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/steinb95/development/lm-polygraph/lm-polygraph/src/lm_polygraph/utils/manager.py", line 534, in calculate
    raise e
  File "/home/steinb95/development/lm-polygraph/lm-polygraph/src/lm_polygraph/utils/manager.py", line 518, in calculate
    new_stats = stat_calculator(
                ^^^^^^^^^^^^^^^^
  File "/home/steinb95/development/lm-polygraph/lm-polygraph/src/lm_polygraph/stat_calculators/sample.py", line 46, in __call__
    temperature=model.parameters.temperature,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'temperature'

I tried a couple of things, but I am simply unclear where to supply the temperature?

Best
P

[psteinb]

So I looked into this further, there must be something in the code, which triggers this problem with the temperature member variable. Because the following snippet works just fine:

import os

import openai


def main():

    openai.api_key = os.environ.get("OPENAI_API_KEY")
    openai.base_url = os.environ.get("OPENAI_API_BASE")

    response = openai.ChatCompletion.create(
        model="Mistral-7B-Instruct-v0.2",
        messages=[{"role": "user", "content": "Tell me what fastapi!"}],
    )
    print(response)


if __name__ == "__main__":
    main()

[rvashurin]

Hi, @psteinb! Thanks for your interest in our library!

Firstly, we do not currently support talking to custom openai-like APIs. It seems like a simple thing to implement though, we will try to do that in a short time.

On the second issue, about the temperature setting. Please note, that parameters option should be an instance of GenerationParameters class, defined in lm_polygraph.utils.generation_parameters module. So in your case is if you want to set temperature other than defaul value of 1.0 you should do like this:

from lm_polygraph.utils.generation_parameters import GenerationParameters

def main():
    params = GenerationParameters(temperature=0.7)
    print(f":: black box test, using Mistral-7B-Instruct-v0.2 from {os.environ["OPENAI_API_BASE"]}")
    model = BlackboxModel(openai_api_key=os.environ["OPENAI_API_KEY"], model_path="Mistral-7B-Instruct-v0.2", parameters=params)

    print(model.parameters)

    print(":: using estimator EigValLaplacian")
    estimator = EigValLaplacian(verbose=True)
    answer = estimate_uncertainty(
        model, estimator, input_text="When did Albert Einstein die?"
    )
    print(">>",answer)

When you pass a dict as parameters option, you replace default config dataclass with it, and polygraph fails trying to get generation config from it as if it was a dataclass.

We will make sure to reflect this in our documentation as we continue to improve it.

Please let me know if you have any other questions.

[rvashurin]

@cant-access-rediska0123 can you try implementing base_url parameter passing to openai library along with api_key?

[IINemo]

@cant-access-rediska0123 could you check that HF wrapper works?

[IINemo]

HF wrapper works. I have checked.
The library now implements access to grey-box deployed by openai API (vLLM, etc).