{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/roformer-enhanced-transformer-with-rotary","title":"RoFormer: Enhanced Transformer with Rotary Position Embedding","arxiv_id":"2104.09864","date":"2021-04-20","proceeding":null,"authors":["Jianlin Su","Yu Lu","Shengfeng Pan","Ahmed Murtadha","Bo Wen","Yunfeng Liu"],"abstract":"Position encoding recently has shown effective in the transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of the sequence. In this paper, we first investigate various methods to integrate positional information into the learning process of transformer-based language models. Then, we propose a novel method named Rotary Position Embedding(RoPE) to effectively leverage the positional information. Specifically, the proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation. Notably, RoPE enables valuable properties, including the flexibility of sequence length, decaying inter-token dependency with increasing relative distances, and the capability of equipping the linear self-attention with relative position encoding. Finally, we evaluate the enhanced transformer with rotary position embedding, also called RoFormer, on various long text classification benchmark datasets. Our experiments show that it consistently overcomes its alternatives. Furthermore, we provide a theoretical analysis to explain some experimental results. RoFormer is already integrated into Huggingface: \\url{https://huggingface.co/docs/transformers/model_doc/roformer}.","url_abs":"https://arxiv.org/abs/2104.09864v5","url_pdf":"https://arxiv.org/pdf/2104.09864v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/ZhuiyiTechnology/roformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/AryaAftab/rotary-embedding-tensorflow","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/airi-institute/gena_lm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/baichuan-inc/baichuan-13b","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/baichuan-inc/baichuan-7b","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/bangoc123/rope-visualization","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/databrickslabs/dolly","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/ezelikman/STaR","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/huggingface/transformers","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/kakaobrain/kogpt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/kingoflolz/mesh-transformer-jax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/lucidrains/reformer-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/lucidrains/rotary-embedding-torch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/singaln/Roformer_Simlarity","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/varungumma/fairseq","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/willGuimont/transformers","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/JunnYu/RoFormer_pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/Mind23-2/MindCode-162","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok"}},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/PaddlePaddle/PaddleNLP/tree/develop/examples/language_model/roformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":null},{"paper_slug":"roformer-enhanced-transformer-with-rotary","repo_url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":null,"task_name":"Position"},{"task_slug":"semantic-text-matching","task_name":"Semantic Text Matching"},{"task_slug":"text-classification","task_name":"Text Classification"}],"methods":[{"method_slug":"rope","method_name":"Rotary Embeddings"}],"datasets_introduced":[],"methods_introduced":[{"slug":"rope","name":"Rotary Embeddings","full_name":"Rotary Position Embedding"}],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2104.09864","atlas_url":"https://app.syntology.ai/?focus=2104.09864","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.09864"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huggingface/transformers","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/databrickslabs/dolly","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/willGuimont/transformers","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ezelikman/STaR","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AryaAftab/rotary-embedding-tensorflow","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/airi-institute/gena_lm","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Mind23-2/MindCode-162","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/baichuan-inc/baichuan-13b","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PaddlePaddle/PaddleNLP/tree/develop/examples/language_model/roformer","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kakaobrain/kogpt","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucidrains/rotary-embedding-torch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucidrains/reformer-pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JunnYu/RoFormer_pytorch","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/ZhuiyiTechnology/roformer","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/varungumma/fairseq","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/singaln/Roformer_Simlarity","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kingoflolz/mesh-transformer-jax","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/baichuan-inc/baichuan-7b","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bangoc123/rope-visualization","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_draft_wrong":5,"ran_honours":1,"ran_fixture":1,"unverified":2},"by_repo_kind":{"listed":{"samples":16,"ran":14,"repositories":10}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ec6ef1c866e39c6f","entry":"MultiHeadsAttention","repo":"singaln/Roformer_Simlarity","repo_kind":"listed","path":"Rotransformer.py","file_url":"https://github.com/singaln/Roformer_Simlarity/blob/HEAD/Rotransformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ec6ef1c866e39c6f"}},{"code_sha256_prefix":"8f063cad604c5b86","entry":"RoPE","repo":"labmlai/annotated_deep_learning_paper_implementations","repo_kind":"listed","path":"labml_nn/neox/model.py","file_url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/HEAD/labml_nn/neox/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8f063cad604c5b86"}},{"code_sha256_prefix":"0fe09c685e79fefc","entry":"RotaryEmbedding","repo":"AryaAftab/rotary-embedding-tensorflow","repo_kind":"listed","path":"rotary_embedding_tensorflow/rotary_embedding_tensorflow.py","file_url":"https://github.com/AryaAftab/rotary-embedding-tensorflow/blob/HEAD/rotary_embedding_tensorflow/rotary_embedding_tensorflow.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0fe09c685e79fefc"}},{"code_sha256_prefix":"dbb6d6a0d27a25f4","entry":"RotaryEmbedding","repo":"varungumma/fairseq","repo_kind":"listed","path":"fairseq/modules/rotary_embedding.py","file_url":"https://github.com/varungumma/fairseq/blob/HEAD/fairseq/modules/rotary_embedding.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbb6d6a0d27a25f4"}},{"code_sha256_prefix":"9a156a004cf5913c","entry":"RotaryEmbedding","repo":"airi-institute/gena_lm","repo_kind":"listed","path":"src/gena_lm/modeling_bert.py","file_url":"https://github.com/airi-institute/gena_lm/blob/HEAD/src/gena_lm/modeling_bert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a156a004cf5913c"}},{"code_sha256_prefix":"0a4292dfdf0c449d","entry":"RotaryEmbedding","repo":"lucidrains/rotary-embedding-torch","repo_kind":"listed","path":"rotary_embedding_torch/rotary_embedding_torch.py","file_url":"https://github.com/lucidrains/rotary-embedding-torch/blob/HEAD/rotary_embedding_torch/rotary_embedding_torch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a4292dfdf0c449d"}},{"code_sha256_prefix":"d05c5eb6cdfb5b97","entry":"RotaryEmbedding","repo":"baichuan-inc/baichuan-7b","repo_kind":"listed","path":"models/modeling_baichuan.py","file_url":"https://github.com/baichuan-inc/baichuan-7b/blob/HEAD/models/modeling_baichuan.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d05c5eb6cdfb5b97"}},{"code_sha256_prefix":"60f6b43d9fa0bb55","entry":"apply_rotary_emb","repo":"varungumma/fairseq","repo_kind":"listed","path":"fairseq/modules/rotary_embedding.py","file_url":"https://github.com/varungumma/fairseq/blob/HEAD/fairseq/modules/rotary_embedding.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60f6b43d9fa0bb55"}},{"code_sha256_prefix":"0a495c8bcfb0380b","entry":"apply_rotary_emb","repo":"lucidrains/rotary-embedding-torch","repo_kind":"listed","path":"rotary_embedding_torch/rotary_embedding_torch.py","file_url":"https://github.com/lucidrains/rotary-embedding-torch/blob/HEAD/rotary_embedding_torch/rotary_embedding_torch.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a495c8bcfb0380b"}},{"code_sha256_prefix":"3c0f5df83f530eba","entry":"apply_rotary_pos_emb","repo":"lucidrains/reformer-pytorch","repo_kind":"listed","path":"reformer_pytorch/reformer_pytorch.py","file_url":"https://github.com/lucidrains/reformer-pytorch/blob/HEAD/reformer_pytorch/reformer_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c0f5df83f530eba"}},{"code_sha256_prefix":"e37b416bd6989338","entry":"relative_positional_encoding","repo":"willGuimont/transformers","repo_kind":"listed","path":"nnet/positional_encoding/relative_positional_encoding.py","file_url":"https://github.com/willGuimont/transformers/blob/HEAD/nnet/positional_encoding/relative_positional_encoding.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e37b416bd6989338"}},{"code_sha256_prefix":"4c8c38feed7983ab","entry":"repeat","repo":"AryaAftab/rotary-embedding-tensorflow","repo_kind":"listed","path":"rotary_embedding_tensorflow/rotary_embedding_tensorflow.py","file_url":"https://github.com/AryaAftab/rotary-embedding-tensorflow/blob/HEAD/rotary_embedding_tensorflow/rotary_embedding_tensorflow.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4c8c38feed7983ab"}},{"code_sha256_prefix":"58823d9435a8751b","entry":"rotate_half","repo":"varungumma/fairseq","repo_kind":"listed","path":"fairseq/modules/rotary_embedding.py","file_url":"https://github.com/varungumma/fairseq/blob/HEAD/fairseq/modules/rotary_embedding.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"58823d9435a8751b"}},{"code_sha256_prefix":"50c38f60f1badef7","entry":"slice_at_dim","repo":"lucidrains/rotary-embedding-torch","repo_kind":"listed","path":"rotary_embedding_torch/rotary_embedding_torch.py","file_url":"https://github.com/lucidrains/rotary-embedding-torch/blob/HEAD/rotary_embedding_torch/rotary_embedding_torch.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"50c38f60f1badef7"}},{"code_sha256_prefix":"c4c1375689d7d3dc","entry":"RelativePositionalEncoding","repo":"willGuimont/transformers","repo_kind":"listed","path":"nnet/positional_encoding/relative_positional_encoding.py","file_url":"https://github.com/willGuimont/transformers/blob/HEAD/nnet/positional_encoding/relative_positional_encoding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c4c1375689d7d3dc"}},{"code_sha256_prefix":"5e8d748cacb045bd","entry":"RoFormerSelfAttention","repo":"JunnYu/RoFormer_pytorch","repo_kind":"listed","path":"src/roformer/modeling_roformer.py","file_url":"https://github.com/JunnYu/RoFormer_pytorch/blob/HEAD/src/roformer/modeling_roformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5e8d748cacb045bd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}