Keep getting this error on local docker run for a local mistral model

[saakethch]

using this
CHECKPOINT = "mistralaiprivate\Mistral-7B-Instruct-v0.1"

class Model:
def init(self, **kwargs):
self._model = None

def load(self):

    self._model = pipeline(
        "text-generation",
        model=CHECKPOINT,
        torch_dtype=torch.bfloat16,
        device_map="auto",
    )
   where mistralaiprivate is the mistral model 10GB model
   Docker log : 
   File \"/usr/local/lib/python3.11/dist-packages/transformers/configuration_utils.py\", line 696, in _get_config_dict\n    raise EnvironmentError(\nOSError: Can't load the configuration of 'mistralaiprivate\\Mistral-7B-Instruct-v0.1'.
   
   I tried all combinations of formating the checkpoint path in the docker images, 
   Could you please help me out 

[squidarth]

Hi @saakethch -- could you elaborate more on what you're trying to do here? The CHECKPOINT that you're referring to should be some repo on HuggingFace. I don't see any org called "mistralaiprivate". (Or anything under mistralaiprivate/Mistral-7B-Instruct-v0.1). There is this however: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1

[saakethch]

HI @squidarth Thanks for replying,
I am actually trying to host a downloaded Mistral Model , i have downloaded the model from huggingface and i am trying to put that in the docker container .
As whenever we initiate the docker the model gets downloaded,
Trying to make this to be a completely local container with the downloaded model inferencing,

[squidarth]

@saakethch I see, so if I understand you correctly you have downloaded the weights to your computer. You are now trying to build them into your Truss, is that right?

Someting that you could do here is put the weights in a folder called data in your Truss. Then, when you create your model, you can reference those, see this guide on how to use Transformers offline so it doesn't try to download them: https://huggingface.co/docs/transformers/main/en/installation#fetch-models-and-tokenizers-to-use-offline

[saakethch]

Thank you,
I have updated my truss example to the latest code pushed.
I have placed my model weights in data/Mistral-7B-Instruct-v0.1
Then i ahve referenced that folder in config.yaml

[squidarth]

Is it working now? You might need to reference the subfolder in “data” rather than just the “data” folder

[saakethch]

@squidarth
Could you please look into this?
When i deployed using truss image build . with the weights in data folded
i get this error in docker :
Could not load model data/ with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>

According to this thread:
databrickslabs/dolly#60

docker log error when I Implementing the fix trust_remote_code : True discussed in this thread -
"The argument trust_remote_code is to be used with Auto classes. It has no effect here and is ignored"

Could you please tell me how my truss code should change so that i can deploy my local weights model on docker using truss.

would help me by a lot, thank you

[saakethch]

I am using the latest truss example code for mistral-chat-7b