Can it work with the Meta Quest 3?
Closed this issue · 7 comments
Describe the feature
The Meta Quest 3 runs on Android, and I’d like to know if it’s possible to run Llama 3 on it with good performance. Also, could microphone compatibility (speech-to-text) be added in future updates? That would be fantastic, I want to create a 3D character that you can interact with in mixed reality using the headset.
I made a very realistic facial animation system for my game and I really want to add Llama 3 to my game. If Llama 3 can't works on meta quest 3 what do you recommend for android ? thanks
Wowwedopti.mp4
Hi, sounds great!
I don't have a Meta Quest so can't tell but if it runs normal Android apps it should work.
You can easily test it by building the AndroidDemo sample (check the "Build a mobile app on Android" section of the readme).
You can try the small and tiny models that can be downloaded from the LLM GameObject.
Let me know if it works!
For speech--to-text you can use whisper.unity.
Ok, I'm going to do some tests. I'll let you know if it works.
Good news, Llama 3 works perfectly on the Quest 3! I managed to get it running. However, I’m having trouble integrating Whisper, it detects the microphone but doesn’t transcribe. I’ll keep looking into it.
Cutllama2.mp4
I found out how to write the text vocally, but I can't figure out how to send/submit the message to the AI when I've finished my sentence :/
I found how to do the speech-to-text, and it works perfectly. However, I'm having a bug with the LLMCharacter. When I talk to it, after about 2 minutes, it starts repeating the same phrase over and over as a response. I'm using a tiny model. Do you know why the AI is repeating a phrase in a loop?
Awesome, thanks for sharing!
You can try to increase the Repeat Penalty or Frequency Penalty of the LLMCharacter (advanced options) - or a different model.