offline huggingface

[Nuclear6]

Because the region is restricted, I would like to ask how to load the model on hugging face offline?

[stevenlu137]

1. You may try to download the model in GGUF format and use local inference services such as ollama/LM studio to solve this problem.

2. If you want to continue using the huggingface model, you can try this:
   add the following configuration to config.json:

   "file": {
   "modelWrapper": "AModelLLAMA",
   "modelList": {
   "/path/to/model": {
   "formatter": "...",
   "contextWindow": ...,
   "systemAsUser": false
   }}
   },

and run it with the following command:

ailice_web --modelID=file:/path/to/model --prompt="main"

This method loads the model from a local path and theoretically does not require a network, but I am not sure if it still work properly under the new version of transformers.

[Nuclear6]

I moved to ubuntu and used micromamba to build an isolation environment. I encountered the following error:

ModuleNotFoundError: No module named 'ailice.core.llm.AModelMistral'

[stevenlu137]

Sorry for that, I forgot to submit two newly created files.
Just get the latest code on master branch and try again. Please let me know if you have any questions

[stevenlu137]

I forgot to remind you that the new version of the code needs to install new dependencies. If you are not reinstalling AIlice, just do this:
pip install mistralai anthropic

[Nuclear6]

not work, this is my config.json

ailice_main --modelID=file:/home/work/.cache/huggingface/hub/models--meta-llama--Llama-2-7b-chat-hf/snapshots/c1b0db933684edbfe29a06fa47eb19cc48025e93 --prompt="main"

[stevenlu137]

My mistake, I forgot that the latest AIlice uses a vector database implementation based on the transformers model. There is currently a temporary solution, modify config.json as follows:

"services": {
"storage": {
"cmd": "python3 -m ailice.modules.AStorageChroma --addr=ipc:///tmp/AIliceStorage.ipc",
"addr": "ipc:///tmp/AIliceStorage.ipc"
},
....
}

Note this is just a temporary solution, AStorageChroma.py will be removed in future. You may need to install chroma in your env.

[Nuclear6]

This time it works. Modify the configuration as you said, and then install the chromadb library. You can see the interactive interface.

pip install chromadb

ailice_main --modelID=file:/home/work/.cache/huggingface/hub/models--meta-llama--Llama-2-7b-chat-hf/snapshots/c1b0db933684edbfe29a06fa47eb19cc48025e93 --prompt="main"

[stevenlu137]

This is a temporary solution. Considering that AIlice's researcher will also use network tools such as Google, and the speech function also requires hf to download the model, it is not practical to run completely off the network. It is recommended to improve your network environment.
I wish you a happy exploration .