{"url":"/method/relative-position-encodings","slug":"relative-position-encodings","name":"Relative Position Encodings","full_name":"Relative Position Encodings","full_name_withheld":false,"description_markdown":"**Relative Position Encodings** are a type of position embeddings for [Transformer-based models](https://paperswithcode.com/methods/category/transformers) that attempts to exploit pairwise, relative positional information. Relative positional information is supplied to the model on two levels: values and keys. This becomes apparent in the two modified self-attention equations shown below. First, relative positional information is supplied to the model as an additional component to the keys\r\n\r\n$$ e\\_{ij} = \\frac{x\\_{i}W^{Q}\\left(x\\_{j}W^{K} + a^{K}\\_{ij}\\right)^{T}}{\\sqrt{d\\_{z}}} $$\r\n\r\nHere $a$ is an edge representation for the inputs $x\\_{i}$ and $x\\_{j}$. The [softmax](https://paperswithcode.com/method/softmax) operation remains unchanged from vanilla self-attention. Then relative positional information is supplied again as a sub-component of the values matrix:\r\n\r\n$$ z\\_{i} = \\sum^{n}\\_{j=1}\\alpha\\_{ij}\\left(x\\_{j}W^{V} + a\\_{ij}^{V}\\right)$$\r\n\r\nIn other words, instead of simply combining semantic embeddings with absolute positional ones, relative positional information is added to keys and values on the fly during attention calculation.\r\n\r\nSource: [Jake Tae](https://jaketae.github.io/study/relative-positional-encoding/)\r\n\r\nImage Source: [Relative Positional Encoding for Transformers with Linear Complexity](https://www.youtube.com/watch?v=qajudaEHuq8","description_state":"present","introduced_year":null,"introduced_by":{"title":"Self-Attention with Relative Position Representations","paper":"/paper/self-attention-with-relative-position","first_author":"Peter Shaw","n_authors":3,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/self-attention-with-relative-position"},"source":{"url":"http://arxiv.org/abs/1803.02155v2","title":"Self-Attention with Relative Position Representations","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Position Embeddings","url":"/methods/category/position-embeddings","pwc_aliases":[]}],"n_papers_tagged":45,"archive_num_papers":45,"papers_newest_first":[{"paper":null,"title":"Two-Player Zero-Sum Games with Bandit Feedback","date":"2025-06-17","arxiv_id":"2506.14518","n_code_links":0,"syntology":null},{"paper":null,"title":"Koopman-Based Event-Triggered Control from Data","date":"2025-04-19","arxiv_id":"2504.14334","n_code_links":0,"syntology":null},{"paper":null,"title":"Performance-Barrier Event-Triggered PDE Control of Traffic Flow","date":"2025-01-01","arxiv_id":"2501.00722","n_code_links":0,"syntology":null},{"paper":null,"title":"Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Model","date":"2024-12-11","arxiv_id":"2412.09647","n_code_links":0,"syntology":null},{"paper":null,"title":"Automated Toll Management System Using RFID and Image Processing","date":"2024-12-02","arxiv_id":"2412.01728","n_code_links":0,"syntology":null},{"paper":"/paper/tulip-token-length-upgraded-clip","title":"TULIP: Token-length Upgraded CLIP","date":"2024-10-13","arxiv_id":"2410.10034","n_code_links":1,"syntology":{"ran":11,"of":20,"unverified":9,"pointer_only":0}},{"paper":null,"title":"Hierarchical Event-Triggered Systems: Safe Learning of Quasi-Optimal Deadline Policies","date":"2024-09-15","arxiv_id":"2409.09812","n_code_links":0,"syntology":null},{"paper":null,"title":"Performance-Barrier Event-Triggered Control of a Class of Reaction-Diffusion PDEs","date":"2024-07-11","arxiv_id":"2407.08178","n_code_links":0,"syntology":null},{"paper":"/paper/liere-generalizing-rotary-position-encodings","title":"LieRE: Generalizing Rotary Position Encodings","date":"2024-06-14","arxiv_id":"2406.10322","n_code_links":1,"syntology":{"ran":1,"of":3,"unverified":2,"pointer_only":0}},{"paper":null,"title":"Contextual Dynamic Pricing: Algorithms, Optimality, and Local Differential Privacy Constraints","date":"2024-06-04","arxiv_id":"2406.02424","n_code_links":0,"syntology":null},{"paper":null,"title":"Event-Triggered Robust Cooperative Output Regulation for a Class of Linear Multi-Agent Systems with an Unknown Exosystem","date":"2024-03-01","arxiv_id":"2403.00645","n_code_links":0,"syntology":null},{"paper":null,"title":"Replication-proof Bandit Mechanism Design with Bayesian Agents","date":"2023-12-28","arxiv_id":"2312.16896","n_code_links":0,"syntology":null},{"paper":null,"title":"Learning-based Scheduling for Information Accuracy and Freshness in Wireless Networks","date":"2023-10-24","arxiv_id":"2310.15705","n_code_links":0,"syntology":null},{"paper":null,"title":"Listen to Minority: Encrypted Traffic Classification for Class Imbalance with Contrastive Pre-Training","date":"2023-08-31","arxiv_id":"2308.16453","n_code_links":0,"syntology":null},{"paper":null,"title":"High-dimensional Contextual Bandit Problem without Sparsity","date":"2023-06-19","arxiv_id":"2306.11017","n_code_links":0,"syntology":null},{"paper":null,"title":"Permutation Decision Trees","date":"2023-06-05","arxiv_id":"2306.02617","n_code_links":0,"syntology":null},{"paper":"/paper/swinia-self-supervised-blind-spot-image","title":"SwinIA: Self-Supervised Blind-Spot Image Denoising without Convolutions","date":"2023-05-09","arxiv_id":"2305.05651","n_code_links":0,"syntology":null},{"paper":null,"title":"An Improved Heart Disease Prediction Using Stacked Ensemble Method","date":"2023-04-12","arxiv_id":"2304.06015","n_code_links":0,"syntology":null},{"paper":null,"title":"Asynchronous Event-Triggered Control for Non-Linear Systems","date":"2022-11-25","arxiv_id":"2211.13846","n_code_links":0,"syntology":null},{"paper":null,"title":"LittleBird: Efficient Faster & Longer Transformer for Question Answering","date":"2022-10-21","arxiv_id":"2210.11870","n_code_links":0,"syntology":null},{"paper":null,"title":"Short Text Pre-training with Extended Token Classification for E-commerce Query Understanding","date":"2022-10-08","arxiv_id":"2210.03915","n_code_links":0,"syntology":null},{"paper":null,"title":"Embed to Control Partially Observed Systems: Representation Learning with Provable Sample Efficiency","date":"2022-05-26","arxiv_id":"2205.13476","n_code_links":0,"syntology":null},{"paper":"/paper/thompson-sampling-for-bandit-learning-in","title":"Thompson Sampling for Bandit Learning in Matching Markets","date":"2022-04-26","arxiv_id":"2204.12048","n_code_links":1,"syntology":null},{"paper":null,"title":"Efficient Localness Transformer for Smart Sensor-Based Energy Disaggregation","date":"2022-03-29","arxiv_id":"2203.16537","n_code_links":0,"syntology":null},{"paper":null,"title":"ETCetera: beyond Event-Triggered Control","date":"2022-03-03","arxiv_id":"2203.01623","n_code_links":0,"syntology":null},{"paper":null,"title":"Formal Analysis of the Sampling Behaviour of Stochastic Event-Triggered Control","date":"2022-02-21","arxiv_id":"2202.10178","n_code_links":0,"syntology":null},{"paper":null,"title":"Chaos and order in event-triggered control","date":"2022-01-12","arxiv_id":"2201.04462","n_code_links":0,"syntology":null},{"paper":"/paper/combining-global-and-local-attention-with","title":"Combining Global and Local Attention with Positional Encoding for Video Summarization","date":"2021-12-01","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"title":"Detecting Extratropical Cyclones of the Northern Hemisphere with Single Shot Detector","date":"2021-12-01","arxiv_id":"2112.01283","n_code_links":0,"syntology":null},{"paper":"/paper/hierarchical-transformers-are-more-efficient","title":"Hierarchical Transformers Are More Efficient Language Models","date":"2021-10-26","arxiv_id":"2110.13711","n_code_links":3,"syntology":{"ran":2,"of":4,"unverified":2,"pointer_only":2}}],"papers_shown":30,"tasks":[{"task":null,"name":"Position","papers":5},{"task":"/task/language-modelling","name":"Language Modelling","papers":3},{"task":"/task/scheduling","name":"Scheduling","papers":3},{"task":"/task/image-generation","name":"Image Generation","papers":2},{"task":"/task/language-modeling","name":"Language Modeling","papers":2},{"task":"/task/machine-translation","name":"Machine Translation","papers":2},{"task":"/task/management","name":"Management","papers":2},{"task":"/task/multi-armed-bandits","name":"Multi-Armed Bandits","papers":2},{"task":"/task/question-answering","name":"Question Answering","papers":2},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":2},{"task":"/task/thompson-sampling","name":"Thompson Sampling","papers":2},{"task":"/task/translation","name":"Translation","papers":2},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":2},{"task":"/task/autonomous-driving","name":"Autonomous Driving","papers":1},{"task":"/task/machine-learning","name":"BIG-bench Machine Learning","papers":1},{"task":"/task/bench2drive","name":"Bench2Drive","papers":1},{"task":"/task/color-image-denoising","name":"Color Image Denoising","papers":1},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":1},{"task":"/task/denoising","name":"Denoising","papers":1},{"task":"/task/diagnostic","name":"Diagnostic","papers":1}],"tasks_shown":20,"n_tasks":56,"usage_by_year":[{"year":"2018","papers":1},{"year":"2020","papers":5},{"year":"2021","papers":12},{"year":"2022","papers":9},{"year":"2023","papers":7},{"year":"2024","papers":8},{"year":"2025","papers":3}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/relative-position-encodings"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}