mozilla-extensions/firefox-voice

Integrate honkling wakeword

ianb opened this issue · 2 comments

ianb commented

My rough understanding of what we need to do:

  • Pull in this branch of Honkling
  • git submodule update --init --recursive

You can also run npm install on that repository, but I don't know if any of those packages are necessary for our use, so may only be part of the initial setup.

If you run npm start you'll get a working page on http://localhost:8000. This is a good example for what we have to include into recorder.html

You can see an example of a different Firefox Voice wakeword backend in #1744:

We don't want to copy in any UI bits of the Honkling example. My vague sense is that honklingController.js is roughly the later we want to work with, but not so much baseController.js (which seems more UI-oriented). There's a high probability we want to slightly rewrite honklingController.

Once we got it working we could pull in honkling as a submodule, but I find that kind of awkward. We do want to easily be able to update versions. But maybe we could checkout honkling into a directory and include a simple script (bin/update-honkling.sh) that copies the needed files into extension/vendor/honkling/ and add those files to the repository. A little like bin/update-static-site.sh

davidak commented

https://github.com/castorini/ww4ff might be related

are you in contact with @ljj7975?

ljj7975 commented

Yeap, I am in.

Thanks @ianb for initiating this process.

First of all, https://github.com/castorini/ww4ff is related as it is used to generate weights (as a js file) that honkling loads.
However, the hey_firefox branch of Honkling already has the trained weights as submodule. I.e for this deployment, you won't need to worry about it.

To elaborate little more about how the code is structured. The main code you want to integrate is in honklingController

For input processing, we were exploring two different libraries, one based on meyda and one based on tfjs.
We are going with meyda version. therefore all you need will be these lines with modified meyda library

Other files that you will need to integrate are basically these three.
Note that melSpectogram file is unnecessary as this is only for tfjs version

I actually found minor bugs in some places so I will be updating this branch little bit.
However, the changes will likely be small won't be changing the code structure.
I will let you know directly on this thread when there is a change. It should be all cleared out by this weekend.

  • 👍1