Light Model Transformer

Light Model Transformer is a light tool that could transform trained tensorflow model into C++ code. The generated code is based on Intel MKLDNN and provides inferface to do inference without any framework, intent to accelerate the inference performance.

Usage:

Install Intel mkl-dnn

install Intel mkl-dnn via script
- python install_mkldnn.py
- See detail by cmd: python install_mkldnn.py -h

Convert model

make sure you have tensorflow installed. (can check it by import tensorflow in python)
prepare tensorflow model in pb format. (need to freeze one if you do not have it, and here assume you have one named frozen.pb)

run the scripts, eg:

# Transform the model to internal representation
python tf2topo.py --input_model_filename=./frozen.pb \
              --weights_file=saved_model/weights.bin \
              --pkl_file=saved_model/weights.pkl \
              --topo_file=saved_model/topo.txt

# Transform to C++ inference code which is based on MKLDNN
python topo2code.py --topo=saved_model/topo.txt

Compile and test generated inference code

compile and test generated code as below:
- cd inference_code
- vi build.sh and make sure the path of MKLDNN_ROOT is correct
- sh build.sh (Note: opencv is needed to compile the code, and it will create an executable file named 'test')
- ./test -W ../saved_model/weights.bin -b 1 -l 100 (Type ./test -H for help)

Integrate generated code to your own project

Please look into inference_code/Main.cpp for how to use the generated code. In general, the inferface looks like:

// Create a Model object
Model model(weight_path, batch_size);

// Do inference by providing input and return the output
output = model.inference(input);

Note:

'Light' means it is a simple implementation, currently only support CNN networks. And even for CNN, many ops are still not supported.
You may also consider OpenVINO(TM) toolkit for inference acceleration.