abi/secret-llama

idea: Support for use in closed networks

Closed this issue · 3 comments

youhogeon commented

It's surprising that even if you download a model online and then switch it offline, it still works properly.
Is there any way to make it work in an offline environment from the beginning? (Save the model in public directory and use that model)

abi commented

Thanks for the suggestion. Definitely a good idea. I think this is already captured by #9

How would you save the model in the first place to the computer? I'm trying to better understand your use case.

youhogeon commented

Our company uses a closed network. All files from external sources must be imported via USB(or an equivalent method).

So, first, I download the wasm file and parameters of model, import them into a closed network, and then temporarily modify the App.tsx file as follows.

However, I hope it will help you set up your model in a better way.

Thank you again for releasing your great code as open source.

const appConfig = webllm.prebuiltAppConfig;
appConfig.model_list = [
  {
    "model_url": "/models/Llama-3-8B-Instruct-q4f16_1-MLC/",
    "model_id": "Llama-3-8B-Instruct-q4f16_1",
    "model_lib_url": '/models/Llama-3-8B-Instruct-q4f16_1-ctx4k_cs1k-webgpu.wasm',
    "vram_required_MB": 4598.34,
    "low_resource_required": true,
  },
]
appConfig.useIndexedDBCache = true;
  • ❤️1
abi commented

Thank you for detailing your use case. Going to merge this issue with #9. The goal is to provide a way to specify the model files if they're already available on the local machine.

  • 👍1