Repo specifically to study Attention/Transformer
Attention/Transformer is now becoming more prevalent in the field of deep learning, to the point where it is now going beyond the orginal task of NLP and now tapping into outperforming CNNs in image based tasks. As a person who didn't find too much interest in NLP, I honeestly didn't care too much for the implications of attetion/transformer. However, there are recent study that have shown that image classifiers that is built with transformer can match/beat traditional classifiers with CNN, as well as the recent breakthrough made with AlphaFold-v2 speculated to use Transformers in their current network, I believe that studying attention and transformer is a really important for all researchers and engineers who are in the field of AI and deep learning.
Even if Transformers completely dominate the field of computer vision, I personally don't believe that CNN will be replaced completely, as I can see a lot of applcations where using Transformer maybe considered an "overkill", just like how in certain tasks, classifical techniques such as Sobel Filters, Gaussian Blurs are used to extract features, and SVM is used to perform final classification. The purpose of this repo is mostly for my personal studying, but I hope that this resource can be used by others who are also needing to study Attention/Transformer in the future.