Sema – A Playground for Live Coding Music and Machine Learning

Sema is a playground where you can rapidly prototype live coding mini-languages for signal synthesis, machine learning and machine listening.

Sema aims to provide an online integrated environment for designing both abstract high-level languages and more powerful low-level languages.

Sema implements a set of core design principles:

Integrated signal engine – In terms language and signal engine integration, there is no conceptual split. Everything is a signal. However, for the sake of modularity, reusability, and a sound architecture, sema's signal engine is implemented by the sema-engine library.
Single sample signal processing – Per-sample sound processing for supporting techniques that use feedback loops, such as physical modelling, reverberation and IIR filtering.
Sample rate transduction – It is simpler to do signal processing with one principal sample rate, the audio rate. Different sample rate requirements of dependent objects can be resolved by upsampling and downsampling, using a transducer. The transducer concept enables us to accommodate a variety of processes with varying sample rates (video, spectral rate, sensors, ML model inference) within a single engine.
Minimal abstractions – There are no high-level abstractions such as buses, synths, nodes, servers, or any language scaffolding in our signal engine. Such abstractions sit within the end-user language design space.

Dependencies

Sema requires the following dependencies to be installed:

Chrome browser or any Chromium-based browser (e.g. Brave, Microsoft Edge, Opera)
Node.js active LTS version (currently v14.4.0). To switch between node versions, you can use nvm.
NPM cli OR Yarn

In order to run, Sema must connect with a project url and api key to a Supabase backend.

You can either install supabase locally by following these instructions, or set up a free hosted project.
Next, follow the steps on how to connect to a supabase backend.

How to build and run the Sema playground on your machine

If you decide to use npm to build sema, you can follow this list of commands:

$ cd sema
$ npm install
$ npm run build
$ npm run dev

If you decide to go with the Yarn package manager instead, you can use the following list of commands:

To use Yarn:

$ cd sema
$ yarn
$ yarn build
$ yarn dev

Once you have sema running as a node application, you can load it on your browser on the following ports

npm run dev, go to http://localhost:5000 on your browser

Hardware acceleration:

Hardware acceleration will have a drastic effect in Tensorflow.js model training speed.

To enable it in Chrome:

Navigate to chrome://settings
Click the Advanced ▼ button at the bottom of the page
In the System section, ensure the Use hardware acceleration when available checkbox is checked (relaunch Chrome for changes to take effect)

To enable in Firefox:

Go to about:preferences
Scroll till you reach the Performance section, or simply search for "performance"
Enable recommended performance settings, this will enable hardware acceleration if and when it is available

Linux Users

Sema uses Web Audio API Audio Worklets. Their performance seems very sensitive to CPU power scaling. If you are experiencing sound quality issues, try setting the CPU governor to performance mode. e.g on Ubuntu,

$ cpupower frequency-set --governor performance

Documentation

Sema's internal documentation aims at supporting the users learning experience. It is integrated within the application and comprises the following sections:

Sema's Wiki documentation aims at supporting contributions. It focuses on how Sema is designed and built:

Contributing

Sema is an open-source project and hopefully the underlying vision, aims and structure will motivate you to contribute to it. Check the following:

Publications

Bernardo, F., Kiefer, C., Magnusson, T. (2021). Assessing the Support for Creativity of a Playground for Live Coding Machine Learning, In: Baalsrud Hauge J., C. S. Cardoso J., Roque L., Gonzalez-Calero P.A. (eds) Entertainment Computing – ICEC 2021. ICEC 2021. Lecture Notes in Computer Science, vol 13056. Springer, Cham. https://doi.org/10.1007/978-3-030-89394-1_38

Bernardo, F., Kiefer, C., Magnusson, T. (2020). A Signal Engine for a Live Coding Language Ecosystem, J. Audio Eng. Soc., vol. 68, no. 10, pp. 756-766. doi: https://doi.org/10.17743/jaes.2020.0016

Bernardo, F., Kiefer, C., Magnusson, T. (2020). Designing for a Pluralist and User-Friendly Live Code Language Ecosystem with Sema. 5th International Conference on Live Coding, University of Limerick, Limerick, Ireland

Bernardo, F., Kiefer, C., Magnusson, T. (2019). An AudioWorklet-based Signal Engine for a Live Coding Language Ecosystem. In Proceedings of Web Audio Conference 2019, Norwegian University of Science and Technology (NTNU), Trondheim, Norway (Best Paper Award at Web Audio Conference 2019)

mimic-sussex/sema