load model failed
twilight0620 opened this issue · 3 comments
I run the container by docker compose up successfully, and then set the model in webui(http://localhost:7860), i can download the models sucessfully: https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/tree/main, there is TheBloke_Llama-2-7B-Chat-GGML/ dir in models dir.
But when i choose the model and click the load button, it has this error:
text-generation-webui | 08:57:39-112718 INFO Loading "TheBloke_Llama-2-7B-Chat-GGML"
text-generation-webui | 08:57:39-152392 ERROR Failed to load the model.
text-generation-webui | Traceback (most recent call last):
text-generation-webui | File "/app/modules/ui_model_menu.py", line 242, in load_model_wrapper
text-generation-webui | shared.model, shared.tokenizer = load_model(selected_model, loader)
text-generation-webui | File "/app/modules/models.py", line 87, in load_model
text-generation-webui | output = load_func_maploader
text-generation-webui | File "/app/modules/models.py", line 247, in llamacpp_loader
text-generation-webui | model_file = list(Path(f'{shared.args.model_dir}/{model_name}').glob('*.gguf'))[0]
text-generation-webui | IndexError: list index out of range
I also try to modify the docker-compose.yml，i add the "--model TheBloke_Llama-2-7B-Chat-GGML" in - EXTRA_LAUNCH_ARGS="--listen --verbose" # Custom launch args (e.g., --model MODEL_NAME), also the same error .
how can i resolve it?
The error shows its looking for *.gguf. Try https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF
Thank you for helping, @mych4nge - It is kind of you to offer the correct link to @twilight0620! I am closing the issue now because it does not relate to the docker image, but rather just to running models in the software generally.
Please note that the model card linked in the original post states:
Important note regarding GGML files.
The GGML format has now been superseded by GGUF. As of August 21st 2023, llama.cpp no longer supports GGML models. Third party clients and libraries are expected to still support it for a time, but many may also drop support.
Please use the GGUF models instead.
Thank you for all.