Live monitoring
mishushakov opened this issue · 8 comments
it would be nice to be able to run a script that will monitor audio and play back the output of neural net
here are some libs, that could help us accomplish this
using BlackHole i was able to reroute sound from my computer to virtual audio interface
since both input and output was rerouted, monitoring had to be done through headphones
this could be very useful for testing the proposed feature
Multi-Output device feature makes it possible to record & monitor at the same time
That all sounds very cool, forgive me for being slow but what will this add from a user standpoint in addition to what a typical DAW provides? Is this for testing the trained model or for recording the initial samples?
the messages above were testing-instructions
this feature would let you try out models live
evaluating models would be as simple as:
python monitor.py --list to list audio devices
python monitor.py --model=ts9 --input=1 --output=1 to record input 1 and playback through output 1
(or something like that)
why would we do this?
to run the plugin you need a DAW and the plugin = complexity
this will run out of the box = simplicity
why should we do this?
this would allow people to plug-and-play without having to install additional software
yeah recording samples is also cool idea
would solve all the conversion shenanigans
👍
there might be as-well is another way to accomplish the goal
you can run Onnx models in Browser
With ONNX.js, web developers can score pre-trained ONNX models directly on browsers with various benefits of reducing server-client communication and protecting user privacy, as well as offering install-free and cross-platform in-browser ML experience
this is like exactly what i was thinking about
lightning has support for onnx built-in
https://pytorch-lightning.readthedocs.io/en/latest/production_inference.html
if we could figure out how to make it work we would be able to offer online experience for evaluating models without any runtime at all
this would be life changing
That would be pretty amazing if we could do it all in a browser. I briefly looked into onnx when I was working on converting the PyTorch models into tensorflow, and it seems like a good framework.
Lots of good ideas to try out here, great work!
i can do the web part but need help saving models as onnx