Methods › General › Position Embeddings › Rotary Embeddings
Rotary Position Embedding
Rotary Embeddings
Introduced by Jianlin Su et al. in RoFormer: Enhanced Transformer with Rotary Position Embedding
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Rotary Position Embedding, or RoPE, is a type of position embedding which encodes absolute positional information with rotation matrix and naturally incorporates explicit relative position dependency in self-attention formulation. Notably, RoPE comes with valuable properties such as flexibility of being expand to any sequence lengths, decaying inter-token dependency with increasing relative distances, and capability of equipping the linear self-attention with relative position encoding.
Papers archive 2025-07-28
8 shown of 8, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
YourMT3+: Multi-instrument Music Transcription with Enhanced Transformer Architectures and Cross-dataset Stem Augmentation 5 Jul 2024 · 1 repository · arXiv:2407.04822
-
Mitigate Position Bias in Large Language Models via Scaling a Single Dimension 4 Jun 2024 · 1 repository · arXiv:2406.02536Syntology ran 16 of 22 samples · 6 unverified
-
Llama 2: Open Foundation and Fine-Tuned Chat Models 18 Jul 2023 · 19 repositories · arXiv:2307.09288Syntology ran 31 of 52 samples · 21 unverified · 16 pointer-only (licence)
-
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness 27 May 2022 · 13 repositories · arXiv:2205.14135Syntology ran 9 of 30 samples · 21 unverified · 1 pointer-only (licence)
-
PaLM: Scaling Language Modeling with Pathways 5 Apr 2022 · 7 repositories · arXiv:2204.02311Syntology ran 30 of 37 samples · 7 unverified
-
Hierarchical Transformers Are More Efficient Language Models 26 Oct 2021 · 3 repositories · arXiv:2110.13711Syntology ran 2 of 4 samples · 2 unverified · 2 pointer-only (licence)
-
Conformer-based End-to-end Speech Recognition With Rotary Position Embedding 13 Jul 2021 · 0 repositories · arXiv:2107.05907
-
RoFormer: Enhanced Transformer with Rotary Position Embedding 20 Apr 2021 · 20 repositories · arXiv:2104.09864Syntology ran 14 of 16 samples · 2 unverified
Tasks archive 2025-07-28
20 shown of 42 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections