Font2vec : Learning font embeddings

About

Our work successfully creates font embeddings that can be used for achieving the following goals:

Identify the fonts effectively from a given font image, and also suggest fonts similar to it.
Provide an interface to alter features of a font (such as thickness, width, alignment, etc). In addition to generating new fonts from the altered embedding, we also suggest similar fonts.
Combine multiple fonts, and visualize the output of the same by interpolating their embeddings.
Unsupervised labelling of fonts by clustering them effectively. This could augment performance of OCR tasks.

Dataset

We generated the dataset from scratch using Google Fonts and extra fonts were scraped from the web.
.ttf and .otf files are used to create 56 * 56 grayscale images were generated

There are two datasets:

2409 font images, generated from Google Fonts.

gdrive_download 1sLe0Oyf4tP-XXG1OYpbTtC5YPbD3aa1Y gfont_ims.zip
unzip gfont_ims.zip

13,674 font images, generated from (Google Fonts + Fonts scraped from the web).

gdrive_download 13-MaBAjnX2WY5xFwayYt73nm01QHywGJ font_ims_all.zip
unzip font_ims_all.zip

function gdrive_download () {
CONFIRM=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate "https://docs.google.com/uc?export=download&id=$1" -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')
wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$CONFIRM&id=$1" -O $2
rm -rf /tmp/cookies.txt
}

Results

Identifying font in image

We pick 5 font images from our test set and find the closest font to each of them.

Interpolations between fonts

Font Clusters

We performed K-means clustering (with k=8) on embeddings of all fonts
We then found the closest font to each cluster center

1D interpolations

Row 1 - Outlined to Solid
Row 2 - Condensed to Regular-Italic
Row 3 - Serif to SansSerif
Row 4 - Monospace to Cursive
Row 5 - Thin to Bold.

2D interpolations

Visualisation

You can visualize the embeddings of all Google fonts here
This used Tensorboard Projector to project 32dim embeddings in 3D and 2D space with PCA and T-SNE
Embeddings were generated by training a VAE on 13k fonts and using the encoder to predict these codes of Google fonts.

Training

After experimenting with RBMs and number of AE architectures, we eventually settled for the following variational autoencoder :
It was trained for nearly 1500 epochs on Nvidia GTX 1050Ti.
Each step took ~350 microseconds and each epoch on the bigger dataset took ~4.7 seconds

thilan7/font2vec