PyTorch implementation of some attentions for Deep Learning Researchers.
Awesome-attentions provides some attentions used in natural language processing using pytorch.
these attentions can used in neural machine translation, speech recognition, image captioning etc...
Attention mechanism allows to attend to different parts of the source sentence at each step of the output generation.
Instead of encoding the input sequence into a single fixed context vector, we let the model learn how to generate a context vector for each output time step.
| Name | Alignment score function | Citation |
|---|---|---|
| Additive | score(s_t, h_i) = v tanh(W[s_t;h_i]) | Bahdanau 2015 |
| Dot-Product | score(s_t, h_i) = s_t · h_i | Luong 2015 |
| Location-Aware | score(s_t, h_i) = w tanh(Ws_t + Vh_i + b) | Chorowski 2015 |
| Scaled Dot-Product | score(s_t, h_i) = s_t · h_i / d_k | Vaswani 2017 |
| Multi-Head | score(Q, K, V) = (head_1, ..., head_n) W | Vaswani 2017 |
If you have any questions, bug reports, and feature requests, please open an issue on Github.
or Contacts sh951011@gmail.com please.
I appreciate any kind of feedback or contribution. Feel free to proceed with small issues like bug fixes, documentation improvement. For major contributions and new features, please discuss with the collaborators in corresponding issues.
@github{
title = {Awesome-attentions},
author = {Soohwan Kim},
publisher = {GitHub},
url = {https://github.com/sooftware/Awesome-attentions},
year = {2020}
}