ISRLU Custom Layer Plugin Table of Contents:
• Description of activation function • What all files does the sample contain • Prerequisites • Running this sample • Known issues • Citations
1.0 Description of activation function: Inverse Square Root Linear unit is a activation function proposed by Brad Carlile and team in 2018[1]. It has a smoothly saturating negative values for negative inputs and the identity for positive inputs. The mathematical function is given as :
Where α is a hyperparamter which controls the output to which it saturates . The primary advantage is that it is able to faster computation ability in comparision to other functions like ELU(Exponential Linear Unit).
ISRLU function is not yet supported in TensorRT and we developed a custom plugin for the same.
2.0 What all files does the sample contain:
ISRLU Model/ISRLU:
o Contains the “MainFile(1).ipynb” which has the code in which the actual implantation is shown .
o “my_model” contains the “saved_model.pb” generated by the model which is given into TensorRT.
o Model_trtfp32 contains the final “saved_model.pb” generated by the TensorRT.
o It also contains all the other files generated during execution.
Plugin: o “IsrluPlugin.cu” is the main CUDA kernel which is called by TensorRT whenever the activation is called o “IsrluPlugin.h” is the header file for the custom plugin(ISRLU)
“ISRLU_model_pytorch.py”(extra file) is the pytorch model implementing the ISRLU activation function.
“TRT_inference_engine_creation.py” : jupyter notebook for generating the build engine.
“trt_graph.pb”(extra) : sample Testing graph.
3.0 Pre-requisites : A jupyter notebook is given which can be run google Colab notebook It is used the integrated way. If user wants to use local system they are expected to have Tensort 4.0+ and CUDA installed in their system.
4.0 Running this sample: For running this sample one can open “MainFile(1).ipynb in collab notebooks and execute the cells
5.0 Known Issues: The user might experience some compatability issues depending upon the TensorRT version they use . In that case user can refer the TensorRT documentation. 7.0 Citations: [1]: arXiv:1710.09967 [cs.LG]