mudler/LocalAI

[EPIC] Model support dashboard (v2)

mudler opened this issue · 9 comments

mudler commented

This epic is a major tracker for all the backends additions that should be part of LocalAI v2 and ongoing efforts.

The objective is to release a v2 which deprecates old models which are now superseded, plus adding a new set. In order to achieve this my idea is to clean up the current state and start pinning dependencies for all the backends which requires specific environment settings (python-based ones).

Some key points:

  • The go-llama backend will be sunset by the c++ version which is directly tied and easier to pinpoint to llama.cpp versions
  • we will deprecate some (if not all) the duplicated backends arch (like falcon,gpt-2, gpt-neo-x, etc..) which are now supported by llama.cpp
  • the python backends will use conda envionments to pin specific versions
  • having a base point for backends for several languages (rust, c++, python, golang, ...)
  • re-order internal layout to accomodate more backends as needed

Some backends will be deprecated as superseded, as such some repositories will be archived (TBD yet).

Backends:

After conda

We need to still test on master:

  • Vall-E-X
  • diffusers
  • bark
  • autogptq
  • exllama
  • vllm

Some rough first-steps required :

Re-layout

After the re-layout we can add new backends listed above without any clashes and:

  • sunset the current llama in favor of llama-cpp
  • #1287
localai-bot commented

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

Issue description:
The issue is related to the LocalAI project and its implementation of the Image endpoint. The user wants to create a cute baby sea otter image with the given prompt, size, and number of images. However, the code provided in the readInput function is not able to parse the input correctly and returns an error.

Proposed solution:
The issue seems to be related to the parsing of the input parameters. The user can try modifying the code in the readInput function to handle the specific format of the prompt, size, and number of images provided by the OpenAI API. Alternatively, they can try using a different API that provides more flexibility in the input format or use a different tool to generate the image.

If the issue persists, the user can also check if there are any updates available for the LocalAI project or report the issue on the GitHub repository to get help from the community.

Sources:

  • 😄1
Aisuko commented

I am thinking here, Should we need to have a outline likes below:

  • C++
    • llama.cpp series LLM, like: Qwen, chatglm)
  • Python
    • [ ]Huggingface
    • autoGPT
  • Rust
    • rustformers
    • candle

If we keep this in mind, we can split the model to the specific backend. For example, I saw the Qwen LLM mentioned their C++ implementation of it "working in the same way as llama.cpp". So, maybe it can be loaded by using llama.cpp. It should be compatible with C++ our backend.

I also suggest to add some labels are related to the series of the backends. We can know a new model can be compatible by our backends.

mudler commented

conda branch was merged in #1144 . I'm looking now into make the llama.cpp backend on par with llama-go and also add llava support to it.

I'm going to refactor and re-layout things in the new backend directory too

Aisuko commented

@mudler thank you for mentioning. Here are some questions(second, and third one) may need you help #1180 (comment). Here what I am thinking is that we can use a tiny model to test the Rust backend features. And make sure everything ok. Maybe we can merge it.

And if everything ok. We can add other LLMs. I have plan to support Llama2(60% finished but it still has an issue), whisper, and also support onnex format.

  • 👍1
mudler commented

Breaking re-layout PR: #1279

kno10 commented

caching/preloading of transformer and similar models, these are currently automatically loaded on startup into /root/.cache/huggingface/. It seems to be enough to set TRANSFORMERS_CACHE in the environment to the models folder, so maybe this only requires a documentation addition.

  • 👍1
muka commented

I may start looking into #1273 while this progresses. What do you think ?

  • 👍1
mudler commented

I may start looking into #1273 while this progresses. What do you think ?

please feel free to go ahead, there are many pieces involved in here, any help is more than appreciated 👍

  • 👍1
mudler commented

caching/preloading of transformer and similar models, these are currently automatically loaded on startup into /root/.cache/huggingface/. It seems to be enough to set TRANSFORMERS_CACHE in the environment to the models folder, so maybe this only requires a documentation addition.

in #1746 I'm taking care of automatically binding the HF cache variables to the models directories if not set already