Hybrid sonification project using machine learning and neural networks.
Some primary videos demonstrating the project can be found here
This project aims to explore the idea of remoteness & isolation. It is a custom made system that works as a digital bridge over interconnected peers across the network (currently works only locally). This project stems from my personal interest using custom made generative systems enacting human interaction and sonification. Modus Operandi:
Each user is assigned a ball object, dragging it X and Y positions are logged as training data of the model of the neural network module.
The generated output can control real time sonfication algorithms which will run independently, but it may also be used as raw material for live coding and thusly expand on other performance possibilities and interactive media making it a highly flexible and versatile project.
At the moment the system uses a model to train a neural network using the X and Y of the user inputs, once the model is trained the system streams out the prediction rates using regression and X and Y values of the cursor position. These rates are then used as raw material for real time sound synthesis algorithms implemented in SuperCollider.
The system provides a hybrid mode of interaction, and can be used in many contexts. It can be installed as an interactive generative system and receice user data from remote users or local participants and can feed any audio system via OSC communication to create real time sonification algorithms.
The system is developed as a cross platform application running on NodeJS, and JavaScript and SuperCollider, and ml5 for the implementation of machine learning capabilities. I am using SC for sonification but any other audio environment able to receive OSC messages can be used.
I would like to express my gratitude to Dan Shiffman and the creative coding communities of P5.js and Ml5.js respectivelly.