Lightning-AI/litgpt

mistralai/Mistral-7B-v0.3 support

karkeranikitha opened this issue · 4 comments

karkeranikitha commented

When will finetuning support for mistralai/Mistral-7B-v0.3 be integrated?

rasbt commented

Sorry, I don't have an ETA at the moment as there are some other things on the list first. By the way, I started it here #1431 but then noticed that there are some new weights (or weight naming conventions) compared to v0.2 that I'd have to look into further to make it work.

karkeranikitha commented

ok thank you.

davmacario commented

noticed that there are some new weights (or weight naming conventions) compared to v0.2

I did notice that too! It is strange, though, as the 'model.safetensors.index.json' file in the huggingface repo uses the usual weight names...

davmacario commented

@rasbt One other thing I noticed is that if the downloading script decides to use Safetensors, it will not download the 'model.safetensors.index.json', as its name does not match "*.safetensors", as specified at this line.

Shouldn't that line contain:

download_files.append("*.safetensors*")

instead (similarly to what done if downloading .bin files here), or is that the desired behavior?

I can put together a tiny PR if needed :)