ValueError: Trying to set a tensor of shape torch.Size([176128, 32]) in "trellis" (which has shape torch.Size([5636096])), this looks incorrect.

Question

ValueError: Trying to set a tensor of shape torch.Size([176128, 32]) in "trellis" (which has shape torch.Size([5636096])), this looks incorrect.

DmitryRedko opened this issue 2 months ago · 8 comments

I am receiving a warning that NVML cannot be initialized, followed by a ValueError when loading a model from Hugging Face. The error message indicates a mismatch in tensor dimensions.

Steps to Reproduce:
Just run the eval_zeroshot.py script with the --hf_path argument pointing to the Hugging Face model path relaxml/Llama-2-7b-QTIP-2Bit.

The requirements are compatible with your requirements.txt file.

Please let me know if you need any further information.

BTW: I tested the quip-sharp package, and it worked without any errors.

Answer 1 · 2024-10-08T12:27:30.000Z

I solved the problem with NVML. The problem with the dimension remains

Answer 2 · 2024-10-08T19:21:43.000Z

There was a bug in one of the commits with the saved tensor shape. I think I fixed it a few weeks ago - try pulling the latest repo. If that doesn't work, I will get around to fixing it in a week or two. The issue is just that some of the models have the trellis saved as a 2D tensor and others have it saved flattened. In the meantime, you can modify this line to be 1D or 2D to patch the problem.

Answer 3 · 2024-10-09T13:15:32.000Z

If you apply flatten to trellis and resave the model, will it solve the problem of tensor dimension mismatch when loading the model from Hugging Face? Or is there some kind of mask that determines how the tensor should be reshaped?
After performing such manipulation, the model loaded and inference started, but I am not sure if all the trellis weights are now in their correct places.

Answer 4 · 2024-10-09T14:49:41.000Z

Flattening the trellis should be sufficient. Which prequantized model are you using? Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Дмитрий Редько ***@***.***> Sent: Wednesday, October 9, 2024 9:15:54 AM To: Cornell-RelaxML/qtip ***@***.***> Cc: Albert Tseng ***@***.***>; Comment ***@***.***> Subject: Re: [Cornell-RelaxML/qtip] ValueError: Trying to set a tensor of shape torch.Size([176128, 32]) in "trellis" (which has shape torch.Size([5636096])), this looks incorrect. (Issue #6) If you apply flatten to trellis and resave the model, will it solve the problem of tensor dimension mismatch when loading the model from Hugging Face? Or is there some kind of mask that determines how the tensor should be reshaped? After performing such manipulation, the model loaded and inference started, but I am not sure if all the trellis weights are now in their correct places. — Reply to this email directly, view it on GitHub<#6 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AH6WZSGXHOSBOSLICS7YUELZ2UUAVAVCNFSM6AAAAABPQRKQOKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBSGMZDEMJYGU>. You are receiving this because you commented.Message ID: ***@***.***>

Answer 5 · 2024-10-10T09:27:46.000Z

https://huggingface.co/relaxml/Llama-2-7b-QTIP-2Bit

This one. By the way, can I get a fine-tuned version from somewhere?

Answer 6 · 2024-10-10T09:30:04.000Z

And also, can I get a fine-tuned version of QUIP# from somewhere?

Answer 7 · 2024-10-15T01:57:29.000Z

The instruct tuned versions should be on huggingface as well. The QuIP# version is here https://huggingface.co/relaxml/Llama-2-7b-chat-E8P-2Bit and the QTIP version is here https://huggingface.co/relaxml/Llama-2-7b-chat-QTIP-2Bit.

Answer 8 · 2024-10-18T17:56:52.000Z

Everything seems to be working fine on my end. Let me know if you are still running into issues with the HF models.