Talk

Let's build a conversational engine so we can talk to our computers! Demo with audio

Is this project useful to you? Give me a ⬆money upvote!⬆

Supported platforms

Right now, we have been testing this on linux + cuda. The project is still at an early stage, and requires a lot of elbow grease to get running. We'll keep on making it better as time goes on!

Changelog

Wed Jun 21 2023

Talk now uses an event based architecture
Set up still isn't straightforward. We'll give this a pass Wed Jun 14 2023
Talk now responds to you.
Breaking change: You're going to have to add piper to your path. See the manual steps

Goals

Runs completely locally
Usuable by my grandmother, if she spoke english
Simple to extend
Discover little HCI hacks
Being able to learn something while driving
Clean up the LLaMa node cpp binding I added in my forked submodule enough to merge into mainline

Installation

The intended audience for this project at the current state is people who are comfortable with hacking things together.

Using bundled bash script (experimental)

chmod 775 build.sh
./build.sh

If you would like to install piper automatically: (this downloads the piper binaries and the default TTS model)

source install_piper.sh true $([ -n "$BASH" ] && echo 1 || echo 2)

WARNING: The bash script will move the existing config.json file to config.json.bkp and create a new one instead.

Dependencies

Node.js v14.15+
piper, a TTS engine. Make sure to add it to your path. This means calling piper from anywhere in your system should work.
graphviz (optional) for displaying the event graph. This is useful for development.

Using manual steps

npm install
Clone the submodules: git submodule init && git submodule update --recursive
Run npm install in whisper.cpp/examples/addon.node
Build & run them (make sure that whisper.cpp & llama.cpp can run)
- cd whisper.cpp && make
- cd llama.cpp && make
In the whisper.cpp git submodule, run:
- npx cmake-js compile --CDWHISPER_CUBLAS="ON" -T whisper-addon -B Release
Note that the above command has --CDWHISPER_CUBLAS=ON. Change that depending on the build parameters you want for your whisper engine. cmake-js can take cmake flags using --CD{The flag you want}. I'm using CUBLAS=ON because I'm on a 3090. Drop it if you're on a macbook.
mv build/Release/* ../bindings/whisper/
Get weights for the next step! I'm using hermes-13b for LLaMa, and whisper tiny.
In llama.cpp git submodule, build and run the server. Check out their README here for steps on how to do that. LLama should be running on local host port 8080 (We'll clean this up and make it easier to run)
Make sure you can run their example curl!
Change config.json to point to the models you downloaded

Running the whole package

Change the config.json to point to record_audio.sh to listen from mic or sample_audio.sh for bundled audio examples
If record_audio.sh is selected, make sure sox package is install in your system. You can install it apt install sox libsox-fmt-all
Read the code! Figure out which button you'll have to press to initiate the response reflex and have the bot respond
(In another shell) ./llama.cpp/build_server/bin/server -m models/llama/nous-hermes-13b.ggmlv3.q4_K_S.bin -c 2048
npm run start

Display graphs

A graphviz file talk.dot will be created when you press ctrl-C.

You can view the graph by running npm run graph, which will plot an svg and open it.

Contributing

Please do

The bindings suck! How do I make them do what i want?

vim ./${llama/whisper}/examples/addon.node/addon.cpp

paramsingh/talk