bananaml/demo-mistral-7b-instruct-v0.1

out of memory

Closed this issue · 0 comments

Out of memory appears when running
my device : 4090 24G
Here is my code :
import requests
messages = {"role": "user",
"content": "",
"max_new_tokens" : 1000
}
send_mistral_playlod = messages
url = "http://0.0.0.0:8888"
response = requests.post(url,json = send_mistral_playlod)
print(response.text)
If I don't use the requset model, it can be inferred that there is no out of memory problem