Implementation of ML5.js used to control a variable font. Using TONE.JS to handle audio data extraction through fft. This example provides all necessary functions to create a model, add sound as input, train the model and run some predictions. Everything happens in the browser, using TENSORFLOW JS behind the scenes for all the machine learning infrastucture.
Using NODE JS for local server (http-server)
$ npm i http-server$ git clone https://github.com/gaelhugo/ml_music_font.git
$ cd ml_music_font
$ http-serverThe goal of that demo is to use a custom NeuralNetwork model to control the complex graphical components of a variable font with music inputs.
Beside the sliders used to set the initial font parameters, there is 3 ways to interact with the experiment:
- If model is not trained yet, click on
spacebarto record data for 6 seconds - click on
enterto train the model - If model is trained, click on
spacebarto listen and run the predictions
Controlling a variable font is a bit different than regular css Rules. First because the way a variable fonts work As you can see, a variables font not only changes its weight and width, but the shape of each letter too. That means that depending on css parameters, the drawing of the letter will adapt itself.
Nice website with interactive variable fonts preview
To offer more transformation options, there are a bunch of custom parameters that can have an effect on the type.
This custom parameters are called AXIS
We can have default predefined axis, such as Weight, Width, Italic, Slant, Optical Size.
But a font designer can also add her/his own axis.
There is a python api to retrieve all available axis for a Variable Font:
$ pip install fonttools
$ ttx -t fvar /path/to/font.fileIt will generate an XML file with all axis, and the parameters' range.
...
<Axis>
<AxisTag>GRAD</AxisTag>
<Flags>0x0</Flags>
<MinValue>88.0</MinValue>
<DefaultValue>88.0</DefaultValue>
<MaxValue>150.0</MaxValue>
<AxisNameID>261</AxisNameID>
</Axis>
...Once these parameters identified, it's easy to access them through simple css styling.
CSS for Variable Fonts (ref)
There is many ways to interact with variable font axis in css. A basic usage example:
/* Set the default values */
:root {
--GRAD: 0;
}
/* Change value for these elements and their children */
.grade-light {
--GRAD: -1;
}
/* Apply whatever value is kept in the CSS variables */
.grade-light {
font-variation-settings: "GRAD" var(--GRAD);
}
/* Animation option*/
@keyframes width-animation {
from {
--WDTH: 25;
}
to {
--WDTH: 151;
}
}Some examples to see the difference with regular css modifications.
The font for the demo : decovar
@font-face {
font-family: "flex";
src: url("../fonts/DecovarAlpha-VF.ttf");
}/* custom axis parameters. Initially set to 0 */
:root {
--custom-WMX2: 0;
--custom-TRMG: 0;
--custom-BLDA: 0;
--custom-TRMD: 0;
--custom-TRMC: 0;
--custom-SKLD: 0;
--custom-TRML: 0;
--custom-SKLA: 0;
--custom-TRMF: 0;
--custom-TRMK: 0;
--custom-BLDB: 0;
--custom-TRMB: 0;
--custom-TRMA: 0;
--custom-SKLB: 0;
--custom-TRME: 0;
}h1 {
font-family: "flex";
font-variation-settings: "WMX2" var(--custom-WMX2), "TRMG" var(--custom-TRMG),
"BLDA" var(--custom-BLDA), "TRMD" var(--custom-TRMD),
"TRMC" var(--custom-TRMC), "SKLD" var(--custom-SKLD),
"TRML" var(--custom-TRML), "SKLA" var(--custom-SKLA),
"TRMF" var(--custom-TRMF), "TRMK" var(--custom-TRMK),
"BLDB" var(--custom-BLDB), "TRMB" var(--custom-TRMB),
"TRMA" var(--custom-TRMA), "SKLB" var(--custom-SKLB),
"TRME" var(--custom-TRME);
}The ML part we are going to use for that demo relays on the javascript librairy ML5.js, a powerful tool based on tensorflow.js, providing various wrappers, that make machine learning in the browser super easy to use.
Library importation
<script src="https://unpkg.com/ml5@0.5.0/dist/ml5.min.js"></script>ML5 is heavily inspired by P5.js in a way that useful javascript functionnalities for creative coders have been wrapped in a "friendly" framework, that will handle all the "not-so-intuitive-js" part. Ml5 does the same with tensorflow.js.
When it comes to build relatively complex machine learning structures, or how to choose a model and how to use it, ML5 provides friendly access to pre-trained models and easy to use functions.
ML5 comes with a preselected choice of models, ready to use, such as mobilenet (image vision), coco SSD (object detection), posenet (body segmentation), styletransfer and even "basic" gan like (pix2pix). (a good part of TFJS models have been wrapped in ML5)
Basically, ML5 provide simple access to "not-so-simple" ML algorithms. With very few coding notion, it provides powerful functionnalities, from creating a model, to train a model and finally run predictions.
(ml5 sketchrnn cat model example)
It provides also some very useful function like features extraction, classification and even custom Neural Network creation.
There is actually 2 type of custom NeuralNet that have been simplified in ML5.
We can either do some classification or regression. The difference between these 2 tasks are the following:
CLASSIFICATION EXAMPLE
REGRESSION EXAMPLE
To control a variable font, and generate unexpected variations in-between predictions, we are going to use the regression task in our model.
- 1: identify/prepare some data to feed the NN
- 2: set your neural network options & initialize your neural network
- 3: add data to the neural network
- 4: normalize your data & train your neural network
- 5: use the trained model to make predictions
For that demo we need to extract audio data from live sound input. To compute audio frequencies and get numerical values out of it, we use Tone.js, a friendly javascript library to handle audio in the browser.
Library importation
<script src="https://cdnjs.cloudflare.com/ajax/libs/tone/14.7.58/Tone.js"></script>Object initialisation
/**
* create an audio analyser object
* The FFT size affects the resolution of the resulting spectra.
* That value must be a power of two in the range 16 to 16384.
*/
this.analyser = new Tone.Analyser({
type: "fft",
size: 1024,
});
/* activate user microphone */
let audSrc = new Tone.UserMedia();
audSrc.connect(this.analyser);
audSrc.open();
/* getting data in a loop */
let signal = this.analyser.getValue();
for (const [i, h] of signal.entries()) {
//... values for each period of time
}/*
* Here is simple settings to set a custom Neural Network with ML5
* task : method for prediction ("classification" || "regression")
* debug : show a information panel while training to see the loss evolution
* inputs : size of inputs data (can be an obect too)
* outputs: size of outputs
* learningRate: The amount that the weights are updated while training ()
*/
const options = {
task: "regression",
debug: true,
inputs: 1024,
outputs: 15,
learningRate: 0.001,
};
//Initialize your neural network
this.nn = ml5.neuralNetwork(options);
For this demo, we are going to use live piano notes as input for our Neural Network. For each note added in the Network, corresponds a font state. The goal is to set several font settings for each note, so when we play them back, the font should retrieve a specific setting. And all values in-between would be interpretation of our Neural Network.
/*
* ML5 function to record data
* inputs : array of numbers or object
* outputs: array of numbers
*/
this.nn.addData(inputs, outputs);Before being able to train your Network feed with values, we need to normalize the data. Here is an interesting article about why is it important to normalize data before training.
/*
* ML5 function to normalize data
* ML5 issue : all values needs to be at least once bigger than zero, otherwise normalisation fail
*/
this.nn.normalizeData();Once datas have been normalized, we can train the model, with some options.
/*
* ML5 function to train a model
* epochs : number total amount of passes through the dataset
* batchSize: number number of data samples processed before the model is updated
*/
const options = {
epochs: 120,
batchSize: 32,
};
this.nn.train(options, callback);- note about Batch Size and Epoch (ref)
What Is the Difference Between Batch and Epoch?
The batch size is a number of samples processed before the model is updated.
The number of epochs is the number of complete passes through the training dataset.
The size of a batch must be more than or equal to one and less than or equal to the number of samples in the training dataset.
The number of epochs can be set to an integer value between one and infinity. You can run the algorithm for as long as you like and even stop it using other criteria besides a fixed number of epochs (...)
There are no magic rules for how to configure these parameters. You must try different values and see what works best for your problem.
When the model is ready, simply call the prediction for a selected input, the model will provide the desired outputs, depending on the pre-set amount. Predictions are received in a Promise, as an array of numbers.
/*
* ML5 function to use the model when it has been set for "regression"
* If the model was set for "classification", the function would have been
* this.nn.classify(inputs, callback);
* inputs : array of numbers or object
*/
this.nn.predict(inputs, callback);In our demo, the output numbers are the desired value for each parameters for the variable font, so we simply assign each one of the prediction to a parameter to update the design of the font.
/*
* Loop through all predicted values and update the css
*/
for (let i = 0; i < result.length; i++) {
const value = result[i].value;
const slider = this.controllers[i].slider;
slider.value = value;
const param = slider.getAttribute("data-param");
document.documentElement.style.setProperty(param, value);
}Ml5js provides also very useful function not implemented in that demo yet. The datas and the model are saved as a json file and the weights are saved as binary files next to the model.
- Saving and Loading data (useful to retrain a existing model)
this.nn.saveData();this.nn.loadData(data, callback);- Saving and Loading trained model (no need to train the model, ready for prediction)
this.nn.save();const modelInfo = {
model: "path/to/model.json",
metadata: "path/to/model_meta.json",
weights: "path/to/model.weights.bin",
};
this.nn.load(modelInfo, callback);Material design animated icons
Variable font can also be used with any kind of drawings with custom axis. ref
Example provided as example in the context of a talk about usage of machine learning used with variable fonts
Please notify the author if you're planning to use it.