Demo doesn't work.

[caiqizh]

Thank you for the amazing framework! Today when I was trying the following codes (simplest demo), I got the error massage saying that:

return UncertaintyOutput(ue[0], input_text, texts[0], model.model_path, estimator.level)
TypeError: UncertaintyOutput.__init__() takes 5 positional arguments but 6 were given

I am wondering whether the framework is ready to use or you are still implementing them?

from lm_polygraph.utils.model import WhiteboxModel
from lm_polygraph.estimators import *
from lm_polygraph.utils.manager import estimate_uncertainty

ue_method = MeanPointwiseMutualInformation()
estimator = SemanticEntropy()

model = WhiteboxModel.from_pretrained(
    "bigscience/bloom-560m",
    device="cuda:0",
)

input_text = "Who is George Bush?"
estimate_uncertainty(model, ue_method, input_text=input_text)

[cant-access-rediska0123]

Thank you for noticing this. We have just fixed this issue in the latest commit.

The framework is generally ready-to-use. Please contact us if you will have any other issues with the code.

[caiqizh]

Thank you for your update. The problem is solved now!

BTW, I find the Semantic Entropy needs quite a lot of CUDA memory. Even my RTX 4090 with 24GB cannot run it. Do you have any suggestion of the memory needed or if there is any other way to run it?

(I tried to set device="cpu", but still get CUDA out of memory...)

[cant-access-rediska0123]

SemanticEntropy unfortunately requires more memory than most of the other methods, as it uses the DeBERTa model to measure NLI between text generations.

We have fixed the issue with getting CUDA OOM when requesting device="cpu" in the latest commit. Now it should only work on the CPU in this case.

To reduce runtime and memory usage, you can try using a lighter DeBERTa model. As a quick fix, you can modify it here (for example, you can replace microsoft/deberta-large-mnli with microsoft/deberta-base-mnli). We will consider making deberta_path a global argument in future releases.

[caiqizh]

Thank you for your rely! I will try that!