Efficient Attentions for Long Document Summarization
125 Citations•2021•
Luyang Huang, Shuyang Cao, Nikolaus Nova Parulian
journal unavailable
Hepos, a novel efficient encoder-decoder attention with head-wise positional strides to effectively pinpoint salient information from the source is proposed, able to process ten times more tokens than existing models that use full attentions.
Abstract
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, Lu Wang. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.