celtera/avendish

Use this to create a large ML corpus of synth sounds and their parameters?

turian opened this issue · 8 comments

turian commented

Is there a simple way to create a large ML corpus (using Python) to generate a large corpus of synth sounds and their parameters? I am trying to learn synth parameters from a given audio file.

jcelerier commented

hi ! I'm sorry, I'm really not an ML expert. this library can be used to easily generate bindings to either existing or new algorithms written in C++, into other languages. If you are doing AI / ML in python the algorithms already exist in e.g. tensorflow, pytorch, etc as far as I know. Paging @ktatar who is an AI expert and will certainly know much better than me :)

turian commented

@jcelerier @ktatar Hey! So I'm not so worried about the ML part.

What I want is a reasonably powerful software synth, where the parameters are in [0, 1] (hopefully in a sensible range, and then just generate a huge dataset of:

(audio, parameters used to generate that audio)

where there are a fixed number of parameters.

ktatar commented

Hey @turian, there is quite a bit of different ways to do that. There is a recent one running on gpu, and I suspect that this one may be suitable for what you would like to do:

https://github.com/torchsynth/torchsynth

Happy to chat on another channel if you would like to discuss further: https://kivanctatar.com/contact

ktatar commented

@turian I just saw that you already published a cool paper building on torchsynth, so I guess I was too late to answer your question. The paper looks cool! I will check it out.

ktatar commented

oh wait, original torchsynth paper is your work too @turian ! I am slightly confused, maybe I got your question wrong. Doesn't torchsynth generate sound-parameter pairs already?

turian commented

@ktatar @jcelerier There are several reasons we are interested in non-torchsynth ways of creating synth corpora:

  1. Transfer learning to new synths and low-shot learning
  2. Synth modules and possible sound design that are not yet part of torchsynth
ktatar commented

There is some open-source c++ synths that are based on known designs such as DX-7. Would that be sth that interests you?

I noticed a python library where you could also import a VST to python, but I haven't tried the feasibility of this yet.

Lastly, I would say the biggest caveat of torchsynth is that it is not made for realtime synthesis, which is a bummer.

I am happy to have a zoom chat, maybe that is easier?

turian commented

@KTatER Happy to zoom yes. Email is lastname at google's email service.

We've using a DX7 emulator in our torchsynth paper and released the dataset. Note that DX7 doesn't have any noise.

I've persuaded the authors of Surge synth to add python binding.

The DAW hosts you might be talking about are: