Get the uncertainty scores without rerun the models

[caiqizh]

Thanks again for your work!

I noticed that in your framework, we need to first run the model then get the uncertainty scores. While it's perfectly fine when using free models, it could be expensive when working with charging APIs like ChatGPT.

Specifically, I'm curious if there's a way to obtain uncertainty measures for previously generated texts without having to rerun the model.

Any information or suggestions you can offer in this regard would be greatly appreciated. I look forward to hearing from you and learning more about this possibility.

[cant-access-rediska0123]

Hello! Thank you for your feedback and for bringing up this issue.

I understand your concern about the cost of running the model several times. Fortunately, there is a way to obtain uncertainty measures for previously generated texts without having to rerun the model.

To achieve this, you can utilize the estimator classes provided in the framework. For example, if you want to use the LexicalSimilarity uncertainty estimator, you can refer to the documentation of the __call__ method of this estimator here.
Here's an example code snippet that should work for you:

from lm_polygraph.estimators import LexicalSimilarity

# Put your 5-10 text samples from ChatGPT generation
samples = [
    'The capital of France is Paris.',
    'Paris is the capital city of France.',
    'The capital of France is Paris.',
    'In France, the capital is Paris.',
    'The capital city of France is Paris.',
]

estimator = LexicalSimilarity('rougeL')
uncertainty = estimator({
    'blackbox_sample_texts': [samples],
})[0]
print(uncertainty)  # -0.7047619047619047

[caiqizh]

Thank you for your reply! It looks great and I will try that!

[caiqizh]

Thank you for providing the codes for the previously generated text! They have been very helpful, and I've successfully used them for Lexical Similarity analysis. I'm planning to test them for other measurements, including NumSets, Degree matrix (Deg), and Eccentricity.

I noticed that these measurements require two additional statistics: semantic_matrix_entail and semantic_matrix_contra. According to the original paper, I know that these are calculated using DeBERTa over generated samples. I'm wondering if there are any short code snippets available to compute these matrices and feed them into the estimator function.

Thanks!

[caiqizh]

Opened a new issue to discuss it.