ExpertGlasses

Expert eyeglasses recommendation system with Generative Adversarial Networks written in Python, 2020.

Description
Installation
Reminder about API-services
Additional features
- Download pretrained GAN models
- Download images for A/B testing
How to use ExpertGlasses
- In code
- GUI
Performance benchmarks
Sources

Description

The task of recommending eyeframes based on people's appearance is quite hard and there are no similar projects available at the moment. This system takes into account over 20 facial attributes, maps them into eyeglasses features with the help of expert module (which consists with over 40 branches). Expert module apply written beforehand rules to get necessary mappings. High interpretability of such approach guaranties user's understanding and loyality towards the system.

To get necessary attributes from the photo this system uses bunch of machine learning algorithms and existing free services, i.e BetaFaceAPI and Face++. Face++ allows to locate bounding boxes of faces in images, while BetaFace uses various classifiers to get most of the secondary-importance features. So to use this repository, you should have access to the internet.

To detect face shape probabilities, iris color, forehead size, jawtype type, skintone the system uses own pretrained convolutional neural networks (CNNs). These models run on your local machine on CPU.

To get all necessary eyeframes attributes, the large dataset (>8k records) of eyeframes was parsed and processed. Because in real life there are not so many eyeframe models in the local shop available, the generation of unique eyewear by given features was implemented. The system use a conditional GAN followed by Super Resolution GAN to create non-existing high-definition images of the eyeframes.

Installation

Requirements:

Python 3.6 or greater
requests
numpy
pandas
opencv_python_headless
scikit_image
tensorflow
Keras_Applications
PySimpleGUI

For the full list of requirements consider to check requirements.txt

To install repository, use the following commands:

Windows:

$ git clone https://github.com/Defasium/expertglasses
$ cd expertglasses/
$ pip3 install -r requirements.txt

Linux:

$ git clone https://github.com/Defasium/expertglasses
$ cd expertglasses/
$ sudo pip3 install -r requirements.txt

Linux inside virtualenv:

$ git clone https://github.com/Defasium/expertglasses
$ cd expertglasses/
$ pip3 install -r requirements.txt

Reminder about API-services

In current version without Face++ api key you can't use Expert Recommender System!

While BetaFace api has the same free api key for everyone with some limitations, allowing you to upload up to 500 images from your IP per day. Face++ utilizes a different strategy, requiring you to create an account on their site. In this account you can generate your own free api key and api secret in the Apps -> API key section:

After that consider changing values of API_KEY and API_SECRET in the faceplusplus.py module:

...
API_KEY = "api_key"
API_SECRET = "api_secret"
RETURN_ATTRIBUTES = "gender,beauty,age,smiling,headpose,ethnicity,skinstatus"
RETURN_LANDMARK = 1
...

Additional features

In the github repository there are no currently some functionality, e.g. pretrained GAN models and best selling eyeglasses. If you want to include this features in the system, you need to download necessary files. Links are listed bellow.

Download pretrained GAN models

Storage Service	Link
	GDrive
	DropBox
	Mega

Download 2 files, cgan.h5 and srgan.h5, and place them in utils/ directory.

Download images for A/B testing

Storage Service	Link
	GDrive
	DropBox
	Mega

Download files, create a directory named abtest/ and extract man/ and woman/ directories inside of it.

How to use ExpertGlasses

At the current state there are only 2 ways of using expert eyeglasses recommender system: in your python code and via GUI.

In code

To use this system, simply import class in your python code:

from expert_backend import ExpertEyeglassesRecommender

After that create an instance of this class with a specified path to the image:

ins = ExpertEyeglassesRecommender('test.jpg')

# by passing a `lang` parameter you can specify language, which will be used at explanation step
ins = ExpertEyeglassesRecommender('test.jpg', lang='en')

Initialization of class may take quite a long time (from 30 second up to 2 minutes).

After initialization recomendations will be completed and to get the top 6 best images of eyeglasses use:

ins.plot_recommendations()

By changing strategy value you can obtain different results:

# in standart strategy feature with the biggest value impacts the results the most
ins.plot_recommendations(strategy='standart')

# in factorized strategy (default value) features are divided into three main groups:
# shape of eyewear, its rim and other features, thus making results more various
ins.plot_recommendations(strategy='factorized')

# in factorized plus strategy every feature greatly influence the final result
# thus providing many varios nonsimilar eyeframes
ins.plot_recommendations(strategy='factorized_plus')

# in color only strategy the resulting images will have the same color as in color vector
ins.plot_recommendations(strategy='color_only')

# shape only strategy's results are similar to standart's one but doesn't take into account color
ins.plot_recommendations(strategy='shape_only')

If you have downloaded abtest directory, you can also try recommending random most popular eyeglasses (so the system don't use any information about your appearance):

# will output 6 randomly chosen eyeglasses 
ins.plot_recommendations(strategy='most_popular')

To get explanation of the system try:

print(ins.description)

To work with new image use:

ins.update_image('test2.jpg')
ins.expert_module()

You can alse use url as an argument:

ins.update_image('https://github.com/Defasium/expertglasses/blob/master/assets/gui.png?raw=true')
ins.expert_module()

To generate unique image with GANs (implying that you have downloaded pretrained models, look Download pretrained GAN models section) use:

image = ins.generate_unique(show=True)

Will generate eyeglasses according to features, plot it with matplotlib if show is True and returns an numpy.ndarray

GUI

Graphical user interface is powered by PySimpleGui framework, which allows fast and simple prototyping for cross-platform systems.

To launch GUI, use the following command from terminal:

$ python gui.py

In the first popup window you should load an image to process, i.e. the fast start, module would automatically calculate all the necessary stuff in background. After that all machine learning models will be loaded in RAM (loading usually takes from 30 seconds up to 2 minutes). When progressbar will be filled, the main interface will appear.

Performance benchmarks

In this section you can find information about current performance of the system, speed benchmarks and so on.

Speed benchmarks were tested on 2 different machines:

Windows 10 with 4 cores 2.3 GHz Intel® Core™ i5-4200U and 4 GB of RAM (low spec)
Linux Ubuntu 16.04 LTS with 4 cores 3.4 GHz Intel® Core™ i7-4770 and 20 GB of RAM (high spec)

The results are listed below:

Action	Time, low spec	Time, high spec	Standart deviation, low spec	Standart deviation, high spec
Image alignment	4.7s	3.9s	2.4s	0.5s
Cached image alignment	62.0ms	39.1ms	55.0ms	5.7ms
BetaFace api request	1.5s	1.6s	0.4s	0.4s
Cached betaFace api request	2.0μs	0.7μs	0.2μs	0.3μs
Extraction of facial attributes	2.7s	2.3s	1.6s	1.2s
Cached extraction of facial attributes	3.0μs	2.4μs	0.1μs	1.8μs
Features translation	0.78s	0.11s	0.62s	0.54s
Cached features translation	15.0μs	3.2μs	12.1μs	1.6μs
Database search with cosine distances	1.1s	0.8s	0.4s	0.4s
Unique eyeglasses generation	4.2s	4.0s	0.2s	1.4s
Initialization of class instance	44.7s	27.2s	1.9s	2.7s

To measure quality of recommendations A/B testing was made with ~50 participants. After that normalized discounted cumulative gain at top 5 items was calculated. The results are shown below:

As you can see, expert recommender system outperforms the results of randomly chosen best selling eyeframes.

Sources

For classifiers this system uses modified version of ShuffleNetV2, implemented by @opconty.

For upscaling generated images this system uses modified version of @krasserm Super Resolution GAN implementation.

Defasium/expertglasses