MoRA, type:6 / RuntimeError: mat1 and mat2 shapes cannot be multiplied
Opened this issue · 1 comments
daebum1994 commented
Please tell me the solution
I'm using a gemma-2-9b-it, meta-llama-2-7b-hf
And My custom dataset.
Version
- torch 2.4.0
- bitsandbytes 0.43.2
- accelerate 0.33.0
Error Message
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/anaconda3/envs/mora/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1603, in _call_impl
[rank0]: result = forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/anaconda3/envs/mora/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 117, in forward
[rank0]: return F.linear(input, self.weight, self.bias)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: RuntimeError: mat1 and mat2 shapes cannot be multiplied (13920x3584 and 248x248)
0%| | 0/76 [00:00<?, ?it/s
kongds commented
Hello, Can you provide the full traceback of error message?