{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/omninet-omnidirectional-representations-from","title":"OmniNet: Omnidirectional Representations from Transformers","arxiv_id":"2103.01075","date":"2021-03-01","proceeding":null,"authors":["Yi Tay","Mostafa Dehghani","Vamsi Aribandi","Jai Gupta","Philip Pham","Zhen Qin","Dara Bahri","Da-Cheng Juan","Donald Metzler"],"abstract":"This paper proposes Omnidirectional Representations from Transformers (OmniNet). In OmniNet, instead of maintaining a strictly horizontal receptive field, each token is allowed to attend to all tokens in the entire network. This process can also be interpreted as a form of extreme or intensive attention mechanism that has the receptive field of the entire width and depth of the network. To this end, the omnidirectional attention is learned via a meta-learner, which is essentially another self-attention based model. In order to mitigate the computationally expensive costs of full receptive field attention, we leverage efficient self-attention models such as kernel-based (Choromanski et al.), low-rank attention (Wang et al.) and/or Big Bird (Zaheer et al.) as the meta-learner. Extensive experiments are conducted on autoregressive language modeling (LM1B, C4), Machine Translation, Long Range Arena (LRA), and Image Recognition. The experiments show that OmniNet achieves considerable improvements across these tasks, including achieving state-of-the-art performance on LM1B, WMT'14 En-De/En-Fr, and Long Range Arena. Moreover, using omnidirectional representation in Vision Transformers leads to significant improvements on image recognition tasks on both few-shot learning and fine-tuning setups.","url_abs":"https://arxiv.org/abs/2103.01075v1","url_pdf":"https://arxiv.org/pdf/2103.01075v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"omninet-omnidirectional-representations-from","repo_url":"https://github.com/lucidrains/omninet-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"de-en","task_name":"de-en"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/language-modelling-on-one-billion-word","task":"Language Modelling","dataset":"One Billion Word","model":"OmniNetT (Large)","rank_in_archive_order":2,"of":27,"metrics":{"Number of params":"100M","PPL":"21.5"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-one-billion-word","task":"Language Modelling","dataset":"One Billion Word","model":"OmniNetP (Large)","rank_in_archive_order":3,"of":27,"metrics":{"Number of params":"100M","PPL":"21.6"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-one-billion-word","task":"Language Modelling","dataset":"One Billion Word","model":"OmniNetB (Large)","rank_in_archive_order":5,"of":27,"metrics":{"PPL":"22"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2014-english-french","task":"Machine Translation","dataset":"WMT2014 English-French","model":"OmniNetP","rank_in_archive_order":17,"of":57,"metrics":{"BLEU score":"42.6"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2014-english-german","task":"Machine Translation","dataset":"WMT2014 English-German","model":"OmniNetP","rank_in_archive_order":17,"of":91,"metrics":{"BLEU score":"29.8"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2017-chinese","task":"Machine Translation","dataset":"WMT2017 Chinese-English","model":"OmniNetP","rank_in_archive_order":3,"of":3,"metrics":{"BLEU":"23.0"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2017-english","task":"Machine Translation","dataset":"WMT2017 English-Finnish","model":"OmniNetP","rank_in_archive_order":1,"of":1,"metrics":{"BLEU":"20.9"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2017-english-french","task":"Machine Translation","dataset":"WMT2017 English-French","model":"OmniNetP","rank_in_archive_order":1,"of":1,"metrics":{"BLEU":"43.1"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2017-english-german","task":"Machine Translation","dataset":"WMT2017 English-German","model":"OmniNetP","rank_in_archive_order":1,"of":1,"metrics":{"BLEU":"29.0"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2017-russian","task":"Machine Translation","dataset":"WMT2017 Russian-English","model":"OmniNetP","rank_in_archive_order":1,"of":1,"metrics":{"BLEU":"36.2"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2103.01075","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}