2024-07-31 |
Correcting Negative Bias in Large Language Models through Negative Attention Score Alignment |
![](https://camo.githubusercontent.com/feec80fb9df1f65132daae4cc83494d14fec9531d2e7915bbd0dba9291dd68c4/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4e656761746976655f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/d41518a87b3118c341e4ed4634a54b444f16c61559829d3f376f1582fbced65c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f61725869762d50617065722d2532334432363931453f6c6f676f3d6172786976) |
2024-07-29 |
Detecting and Understanding Vulnerabilities in Language Models via Mechanistic Interpretability |
![](https://camo.githubusercontent.com/724a1698f2dc10dfc16f7a20e1fae0dce4ccd89f7e5c43b0db8c624015a2e1cf/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f56756c6e657261626c655f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/d41518a87b3118c341e4ed4634a54b444f16c61559829d3f376f1582fbced65c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f61725869762d50617065722d2532334432363931453f6c6f676f3d6172786976) |
2024-07-22 |
RazorAttention: Efficient KV Cache Compression Through Retrieval Heads |
![](https://camo.githubusercontent.com/8e525832e3a74ac0ec925dcbc433813d9c865dbbfaf528fe6d7c21a462f6d5a7/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f52657472696576616c5f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/d41518a87b3118c341e4ed4634a54b444f16c61559829d3f376f1582fbced65c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f61725869762d50617065722d2532334432363931453f6c6f676f3d6172786976) |
2024-07-21 |
Answer, Assemble, Ace: Understanding How Transformers Answer Multiple Choice Questions |
![](https://camo.githubusercontent.com/32024ac4c54fc30b61c8feb2e61ded64f99d824625114d575b998ebf683f83fa/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f416e737765725f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/d41518a87b3118c341e4ed4634a54b444f16c61559829d3f376f1582fbced65c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f61725869762d50617065722d2532334432363931453f6c6f676f3d6172786976) |
2024-07-09 |
Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning |
![](https://camo.githubusercontent.com/6afd93c5e3a1db6d78f9c5651a8b5dca982eb105a0492665ce4d049d00aa706c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f496e64756374696f6e5f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/d41518a87b3118c341e4ed4634a54b444f16c61559829d3f376f1582fbced65c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f61725869762d50617065722d2532334432363931453f6c6f676f3d6172786976) |
2024-07-01 |
Steering Large Language Models for Cross-lingual Information Retrieval |
![](https://camo.githubusercontent.com/2ab1a1c4ca355cfb114d782c39ef62749dc7b205186a521b0756624605dfbdc0/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f436f686572656e63655f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/16a03183e2eb4875322555d04a8a1a068ab41e7fb59ea097c3cd0049aad607ac/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f53494749522d50617065722d253233443236393145) |
2024-06-21 |
MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression |
![](https://camo.githubusercontent.com/6d309c0c62cbc14fbe9416b460407072973bab6f8f9fa714f8f79940fa8422fe/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4c6f63616c2d2d636f6e746578745f486561642d626c7565) |
![Code](https://camo.githubusercontent.com/770a7d27e46c77aaf2237c83cbe0277cd279f8afbc5ede96529972404bbdf7ca/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4769744875622d436f64652d627269676874677265656e3f6c6f676f3d676974687562) |
2024-06-19 |
On the Difficulty of Faithful Chain-of-Thought Reasoning in Large Language Models |
![](https://camo.githubusercontent.com/1121ac3ded4aa7c70ddf95cc689c5f790411022af6a10a4807f0dbac8a579233/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f466169746866756c6e6573735f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/23515641fda5dc95f812dece5f57c2020ab59725d9925e7a0a56af247e7e4616/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f49434d4c2d50617065722d253233443236393145) |
2024-06-16 |
Induction Heads as a Primary Mechanism for Pattern Matching in In-context Learning |
![](https://camo.githubusercontent.com/6afd93c5e3a1db6d78f9c5651a8b5dca982eb105a0492665ce4d049d00aa706c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f496e64756374696f6e5f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/6686afe4ce736eb3a75fe1e670238e18978b56c56590819ff7838f51494d7488/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4f70656e5265766965772d50617065722d253233443236393145) |
2024-06-04 |
Iteration Head: A Mechanistic Study of Chain-of-Thought |
![](https://camo.githubusercontent.com/12285284c517ef93ad07cfbe79977c389d510e44136bdd62580d1a010d5c4aec/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f497465726174696f6e5f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/d41518a87b3118c341e4ed4634a54b444f16c61559829d3f376f1582fbced65c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f61725869762d50617065722d2532334432363931453f6c6f676f3d6172786976) |
2024-05-28 |
Knowledge Circuits in Pretrained Transformers |
![](https://camo.githubusercontent.com/12ba46caaeb04f184d8416d06332bca4105e35e0ea69c05cc8c94b5166be76bf/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f52656c6174696f6e5f486561642d626c7565) |
![Code](https://camo.githubusercontent.com/770a7d27e46c77aaf2237c83cbe0277cd279f8afbc5ede96529972404bbdf7ca/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4769744875622d436f64652d627269676874677265656e3f6c6f676f3d676974687562) |
2024-05-23 |
Linking In-context Learning in Transformers to Human Episodic Memory |
![](https://camo.githubusercontent.com/6afd93c5e3a1db6d78f9c5651a8b5dca982eb105a0492665ce4d049d00aa706c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f496e64756374696f6e5f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/d41518a87b3118c341e4ed4634a54b444f16c61559829d3f376f1582fbced65c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f61725869762d50617065722d2532334432363931453f6c6f676f3d6172786976) |
2024-05-07 |
How does GPT-2 Predict Acronyms? Extracting and Understanding a Circuit via Mechanistic Interpretability |
![](https://camo.githubusercontent.com/49e1b16a054fe7cf98c3a9817cbdef55f6e702ade7c44c581121ec4a32e7b08c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4c65747465725f4d6f7665725f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/d41518a87b3118c341e4ed4634a54b444f16c61559829d3f376f1582fbced65c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f61725869762d50617065722d2532334432363931453f6c6f676f3d6172786976) |
2024-05-02 |
What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation |
![](https://camo.githubusercontent.com/6afd93c5e3a1db6d78f9c5651a8b5dca982eb105a0492665ce4d049d00aa706c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f496e64756374696f6e5f486561642d626c7565) |
![Code](https://camo.githubusercontent.com/770a7d27e46c77aaf2237c83cbe0277cd279f8afbc5ede96529972404bbdf7ca/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4769744875622d436f64652d627269676874677265656e3f6c6f676f3d676974687562) |
2024-04-24 |
Retrieval Head Mechanistically Explains Long-Context Factuality |
![](https://camo.githubusercontent.com/8e525832e3a74ac0ec925dcbc433813d9c865dbbfaf528fe6d7c21a462f6d5a7/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f52657472696576616c5f486561642d626c7565) |
![Code](https://camo.githubusercontent.com/770a7d27e46c77aaf2237c83cbe0277cd279f8afbc5ede96529972404bbdf7ca/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4769744875622d436f64652d627269676874677265656e3f6c6f676f3d676974687562) |
2024-03-27 |
Non-Linear Inference Time Intervention: Improving LLM Truthfulness |
![](https://camo.githubusercontent.com/02519bd7b748597752ad1976c9cbb9d4485e2a5799745648548eadce6428176c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f547275746866756c6e6573735f486561642d626c7565) |
![Code](https://camo.githubusercontent.com/770a7d27e46c77aaf2237c83cbe0277cd279f8afbc5ede96529972404bbdf7ca/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4769744875622d436f64652d627269676874677265656e3f6c6f676f3d676974687562) |
2024-02-28 |
Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models |
![](https://camo.githubusercontent.com/39c487e9c7fd985e237eba3c8ec26a68397be90a7e09be85844af420f30abc45/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f436f6e746578745f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/d41518a87b3118c341e4ed4634a54b444f16c61559829d3f376f1582fbced65c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f61725869762d50617065722d2532334432363931453f6c6f676f3d6172786976) |
2024-02-27 |
Information Flow Routes: Automatically Interpreting Language Models at Scale |
![](https://camo.githubusercontent.com/11d3848aa9a0431ad9842ba336ce344c3a1efbc0c31398067fef8d2bc896c9ab/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f537562776f72645f6d657267696e675f486561642d626c7565) |
![Code](https://camo.githubusercontent.com/770a7d27e46c77aaf2237c83cbe0277cd279f8afbc5ede96529972404bbdf7ca/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4769744875622d436f64652d627269676874677265656e3f6c6f676f3d676974687562) |
2024-02-20 |
Identifying Semantic Induction Heads to Understand In-Context Learning |
![](https://camo.githubusercontent.com/6afd93c5e3a1db6d78f9c5651a8b5dca982eb105a0492665ce4d049d00aa706c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f496e64756374696f6e5f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/d41518a87b3118c341e4ed4634a54b444f16c61559829d3f376f1582fbced65c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f61725869762d50617065722d2532334432363931453f6c6f676f3d6172786976) |
2024-02-16 |
The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains |
![](https://camo.githubusercontent.com/6afd93c5e3a1db6d78f9c5651a8b5dca982eb105a0492665ce4d049d00aa706c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f496e64756374696f6e5f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/d41518a87b3118c341e4ed4634a54b444f16c61559829d3f376f1582fbced65c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f61725869762d50617065722d2532334432363931453f6c6f676f3d6172786976) |
2024-02-11 |
Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMs |
![](https://camo.githubusercontent.com/12ba46caaeb04f184d8416d06332bca4105e35e0ea69c05cc8c94b5166be76bf/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f52656c6174696f6e5f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/d41518a87b3118c341e4ed4634a54b444f16c61559829d3f376f1582fbced65c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f61725869762d50617065722d2532334432363931453f6c6f676f3d6172786976) |
2024-02-05 |
How do Large Language Models Learn In-Context? Query and Key Matrices of In-Context Heads are Two Towers for Metric Learning |
![](https://camo.githubusercontent.com/000e0c534fe8bf9821ac86609659738a5720c54a3e5e0a2f983e1c22baf820da/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f496e2d2d436f6e746578745f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/d41518a87b3118c341e4ed4634a54b444f16c61559829d3f376f1582fbced65c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f61725869762d50617065722d2532334432363931453f6c6f676f3d6172786976) |
2024-01-16 |
Circuit Component Reuse Across Tasks in Transformer Language Models |
![](https://camo.githubusercontent.com/ef11b4fbf63970ca69faedae877191fe611d8e74e5fc2c3e11495f11f428f03c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f436f6e74656e745f47617468657265725f486561642d626c7565) |
![Code](https://camo.githubusercontent.com/770a7d27e46c77aaf2237c83cbe0277cd279f8afbc5ede96529972404bbdf7ca/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4769744875622d436f64652d627269676874677265656e3f6c6f676f3d676974687562) |
2024-01-16 |
Successor Heads: Recurring, Interpretable Attention Heads In The Wild |
![](https://camo.githubusercontent.com/88ddef1f142cbe81d78f23129ff2befbcb47c6441fa72cb4f18904bba4e47c7c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f537563636573736f725f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/af3079dab69dcb8adaf346aad7c6f7fafe63d11d5799a0d2202f389b010be3af/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f49434c522d506f737465722d253233443236393145) |
2024-01-16 |
Function Vectors in Large Language Models |
![](https://camo.githubusercontent.com/3bcb79802bc0dad4cf005fa1abc94e660b8e4d077a571b78b79d63475a846536/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f46756e6374696f6e5f566563746f725f486561642d626c7565) |
![Data](https://camo.githubusercontent.com/140b22e715186cdea6448d3bf02e055fe1b9941821e39b4e1073ebe20e12a928/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4769744875622d446174612d627269676874677265656e3f6c6f676f3d676974687562) |
2023-10-23 |
Linear Representations of Sentiment in Large Language Models |
![](https://camo.githubusercontent.com/dc18484a3ed356fffcd8e3d032bf2d052353ad05b4df0f6d14d5f47795a709c5/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4469726563745f6566666563745f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/d41518a87b3118c341e4ed4634a54b444f16c61559829d3f376f1582fbced65c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f61725869762d50617065722d2532334432363931453f6c6f676f3d6172786976) |
2023-10-06 |
Copy Suppression: Comprehensively Understanding an Attention Head |
![](https://camo.githubusercontent.com/69a6b538c4be3d5900ca5351532b9f6a773c88837546860138fbf0aa6bda6be9/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f436f70795f5375707072657373696f6e5f486561642d626c7565) |
![Demo](https://camo.githubusercontent.com/c19e2fa6d4d8d40f5872be84c6ebe52b715dc9d934253bf130e4ee8e26252ab3/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f44656d6f2d566965772d707572706c653f6c6f676f3d696e7465726e65742d6578706c6f726572) |
2023-09-22 |
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model |
![](https://camo.githubusercontent.com/02519bd7b748597752ad1976c9cbb9d4485e2a5799745648548eadce6428176c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f547275746866756c6e6573735f486561642d626c7565) |
![Code](https://camo.githubusercontent.com/770a7d27e46c77aaf2237c83cbe0277cd279f8afbc5ede96529972404bbdf7ca/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4769744875622d436f64652d627269676874677265656e3f6c6f676f3d676974687562) |
2023-09-22 |
Birth of a Transformer: A Memory Viewpoint |
![](https://camo.githubusercontent.com/6afd93c5e3a1db6d78f9c5651a8b5dca982eb105a0492665ce4d049d00aa706c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f496e64756374696f6e5f486561642d626c7565) |
![Code](https://camo.githubusercontent.com/770a7d27e46c77aaf2237c83cbe0277cd279f8afbc5ede96529972404bbdf7ca/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4769744875622d436f64652d627269676874677265656e3f6c6f676f3d676974687562) |
2023-07-18 |
Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla |
![](https://camo.githubusercontent.com/181459a9293feecf456a16ba3a940eafbe41809c8b487373c3c3ce3643f4d92f/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f53696e676c655f4c65747465725f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/d41518a87b3118c341e4ed4634a54b444f16c61559829d3f376f1582fbced65c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f61725869762d50617065722d2532334432363931453f6c6f676f3d6172786976) |
2023-02-02 |
Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small |
![](https://camo.githubusercontent.com/cfd486f163975aaf4b9db5839ac45bed43aaa4068686fa9ece97654bc213c938/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4261636b75705f4e616d655f4d6f7665725f486561642d626c7565) |
![Code](https://camo.githubusercontent.com/770a7d27e46c77aaf2237c83cbe0277cd279f8afbc5ede96529972404bbdf7ca/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4769744875622d436f64652d627269676874677265656e3f6c6f676f3d676974687562) |
2022-03-08 |
In-context Learning and Induction Heads |
![](https://camo.githubusercontent.com/6afd93c5e3a1db6d78f9c5651a8b5dca982eb105a0492665ce4d049d00aa706c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f496e64756374696f6e5f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/ae3dbba30705093a18842e2a66137d2a694c25deb1a0aa91da19de21986f5aa5/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f416e7468726f7069632d50617065722d253233443236393145) |
2021-12-22 |
A Mathematical Framework for Transformer Circuits |
![](https://camo.githubusercontent.com/6afd93c5e3a1db6d78f9c5651a8b5dca982eb105a0492665ce4d049d00aa706c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f496e64756374696f6e5f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/ae3dbba30705093a18842e2a66137d2a694c25deb1a0aa91da19de21986f5aa5/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f416e7468726f7069632d50617065722d253233443236393145) |
2021-05-18 |
The Heads Hypothesis: A Unifying Statistical Approach Towards Understanding Multi-Headed Attention in BERT |
![](https://camo.githubusercontent.com/8c30b536c3ca12f4b776f88b8e3e2c18eaba7b1f537a393c7a347335e9cfe45d/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f426c6f636b5f486561642d626c7565) |
![Code](https://camo.githubusercontent.com/770a7d27e46c77aaf2237c83cbe0277cd279f8afbc5ede96529972404bbdf7ca/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4769744875622d436f64652d627269676874677265656e3f6c6f676f3d676974687562) |
2021-04-01 |
Have Attention Heads in BERT Learned Constituency Grammar? This paper investigates whether the attention heads in BERT and RoBERTa language models have learned constituency grammar. The researchers use an unsupervised method to extract constituency parsing trees from the attention weights of these models. |
![](https://camo.githubusercontent.com/5a731cee3e093f35c8a9fb1f71539915ca90318acd7d220fe6dbd107e078acad/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f436f6e7374697475656e63795f6772616d6d61725f696e647563696e675f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/177e1870471144b319a748feb693ac7e11a7a03353fa99a5301f66ce6716033c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f41434c2d50617065722d253233443236393145) |
2019-11-27 |
Do Attention Heads in BERT Track Syntactic Dependencies? The researchers investigate if the attention heads in pre-trained transformer language models like BERT and RoBERTa can capture syntactic dependency relations between words. |
![](https://camo.githubusercontent.com/b1b2c9557ce40e842a3888889a3867f5926ea986eb7b418779259d8a8f741d5d/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f53796e7461637469635f646570656e64656e63795f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/d41518a87b3118c341e4ed4634a54b444f16c61559829d3f376f1582fbced65c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f61725869762d50617065722d2532334432363931453f6c6f676f3d6172786976) |
2019-11-01 |
Adaptively Sparse Transformers |
![](https://camo.githubusercontent.com/a9bed3afcb8d0a88b4e889d66704643dd41d2544201c54cf7c3ad0de5795c78f/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f496e746572726f676174696f6e5f486561642d626c7565) |
![Code](https://camo.githubusercontent.com/770a7d27e46c77aaf2237c83cbe0277cd279f8afbc5ede96529972404bbdf7ca/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4769744875622d436f64652d627269676874677265656e3f6c6f676f3d676974687562) |
2019-08-01 |
What does BERT look at? An Analysis of BERT’s Attention |
![](https://camo.githubusercontent.com/05ecd3e39d3a99088fe6b1d990989b22dee17effc464fcdac8a61a837efed873/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f53796e7461637469635f486561642d626c7565) |
![Code](https://camo.githubusercontent.com/770a7d27e46c77aaf2237c83cbe0277cd279f8afbc5ede96529972404bbdf7ca/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4769744875622d436f64652d627269676874677265656e3f6c6f676f3d676974687562) |
2019-05-22 |
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned |
![](https://camo.githubusercontent.com/476bb4af2ed455acaf66f772f396c0be62aff3c0462e93e013cc2102f075c468/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f526172655f776f7264735f486561642d626c7565) |
![Code](https://camo.githubusercontent.com/770a7d27e46c77aaf2237c83cbe0277cd279f8afbc5ede96529972404bbdf7ca/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4769744875622d436f64652d627269676874677265656e3f6c6f676f3d676974687562) |
2016-03-21 |
Incorporating Copying Mechanism in Sequence-to-Sequence Learning |
![](https://camo.githubusercontent.com/8e525832e3a74ac0ec925dcbc433813d9c865dbbfaf528fe6d7c21a462f6d5a7/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f52657472696576616c5f486561642d626c7565) |
![Paper](https://camo.githubusercontent.com/177e1870471144b319a748feb693ac7e11a7a03353fa99a5301f66ce6716033c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f41434c2d50617065722d253233443236393145) |