{"url":"/method/favor","slug":"favor","name":"FAVOR+","full_name":"Fast Attention Via Positive Orthogonal Random Features","full_name_withheld":false,"description_markdown":"**FAVOR+**, or **Fast Attention Via Positive Orthogonal Random Features**, is an efficient attention mechanism used in the [Performer](https://paperswithcode.com/method/performer) architecture which leverages approaches such as kernel methods and random features approximation for approximating [softmax](https://paperswithcode.com/method/softmax) and Gaussian kernels. \r\n\r\nFAVOR+ works for attention blocks using matrices $\\mathbf{A} \\in \\mathbb{R}^{L×L}$ of the form $\\mathbf{A}(i, j) = K(\\mathbf{q}\\_{i}^{T}, \\mathbf{k}\\_{j}^{T})$, with $\\mathbf{q}\\_{i}/\\mathbf{k}\\_{j}$ standing for the $i^{th}/j^{th}$ query/key row-vector in $\\mathbf{Q}/\\mathbf{K}$ and kernel $K : \\mathbb{R}^{d } × \\mathbb{R}^{d} \\rightarrow \\mathbb{R}\\_{+}$ defined for the (usually randomized) mapping: $\\phi : \\mathbb{R}^{d } → \\mathbb{R}^{r}\\_{+}$ (for some $r > 0$) as:\r\n\r\n$$K(\\mathbf{x}, \\mathbf{y}) = E[\\phi(\\mathbf{x})^{T}\\phi(\\mathbf{y})] $$\r\n\r\nWe call $\\phi(\\mathbf{u})$ a random feature map for $\\mathbf{u} \\in \\mathbb{R}^{d}$ . For $\\mathbf{Q}^{'}, \\mathbf{K}^{'} \\in \\mathbb{R}^{L \\times r}$ with rows given as $\\phi(\\mathbf{q}\\_{i}^{T})^{T}$ and $\\phi(\\mathbf{k}\\_{i}^{T})^{T}$  respectively, this leads directly to the efficient attention mechanism of the form:\r\n\r\n$$ \\hat{Att\\_{\\leftrightarrow}}\\left(\\mathbf{Q}, \\mathbf{K}, \\mathbf{V}\\right) = \\hat{\\mathbf{D}}^{-1}(\\mathbf{Q^{'}}((\\mathbf{K^{'}})^{T}\\mathbf{V}))$$\r\n\r\nwhere\r\n\r\n$$\\mathbf{\\hat{D}} = \\text{diag}(\\mathbf{Q^{'}}((\\mathbf{K^{'}})\\mathbf{1}\\_{L})) $$\r\n\r\nThe above scheme constitutes the [FA](https://paperswithcode.com/method/dfa)-part of the FAVOR+ mechanism. The other parts are achieved by:\r\n\r\n- The R part :  The softmax kernel is approximated though trigonometric functions, in the form of a regularized softmax-kernel SMREG, that employs positive random features (PRFs).\r\n- The OR+ part : To reduce the variance of the estimator, so we can use a smaller number of random features, different samples are entangled to be exactly orthogonal using the Gram-Schmidt orthogonalization procedure.\r\n\r\nThe details are quite technical, so it is recommended you read the paper for further information on these steps.","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"https://arxiv.org/abs/2009.14794v4","title":"Rethinking Attention with Performers","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Attention Mechanisms","url":"/methods/category/attention-mechanisms","pwc_aliases":["attention-mechanisms-1"]}],"n_papers_tagged":102,"archive_num_papers":null,"papers_newest_first":[{"paper":null,"title":"ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists","date":"2025-06-02","arxiv_id":"2506.01241","n_code_links":0,"syntology":null},{"paper":null,"title":"WirelessMathBench: A Mathematical Modeling Benchmark for LLMs in Wireless Communications","date":"2025-05-20","arxiv_id":"2505.14354","n_code_links":0,"syntology":null},{"paper":null,"title":"CacheFormer: High Attention-Based Segment Caching","date":"2025-04-18","arxiv_id":"2504.13981","n_code_links":0,"syntology":null},{"paper":null,"title":"Deconstructing Jazz Piano Style Using Machine Learning","date":"2025-04-07","arxiv_id":"2504.05009","n_code_links":0,"syntology":null},{"paper":"/paper/predicting-survivability-of-cancer-patients","title":"Predicting Survivability of Cancer Patients with Metastatic Patterns Using Explainable AI","date":"2025-04-07","arxiv_id":"2504.06306","n_code_links":1,"syntology":null},{"paper":null,"title":"Forecasting Empty Container availability for Vehicle Booking System Application","date":"2025-03-14","arxiv_id":"2503.11728","n_code_links":0,"syntology":null},{"paper":"/paper/stead-spatio-temporal-efficient-anomaly-1","title":"STEAD: Spatio-Temporal Efficient Anomaly Detection for Time and Compute Sensitive Applications","date":"2025-03-11","arxiv_id":"2503.07942","n_code_links":1,"syntology":null},{"paper":null,"title":"Deep Learning-Based Approach for Automatic 2D and 3D MRI Segmentation of Gliomas","date":"2025-02-27","arxiv_id":"2502.19760","n_code_links":0,"syntology":null},{"paper":null,"title":"On the use of Performer and Agent Attention for Spoken Language Identification","date":"2025-02-09","arxiv_id":"2502.05841","n_code_links":0,"syntology":null},{"paper":null,"title":"Nick Patrick Contreras","date":"2025-01-09","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"title":"Comparative Study of Deep Learning Architectures for Textual Damage Level Classification","date":"2025-01-03","arxiv_id":"2501.01694","n_code_links":0,"syntology":null},{"paper":"/paper/electra-and-gpt-4o-cost-effective-partners","title":"ELECTRA and GPT-4o: Cost-Effective Partners for Sentiment Analysis","date":"2024-12-29","arxiv_id":"2501.00062","n_code_links":1,"syntology":null},{"paper":null,"title":"Music Genre Classification: Ensemble Learning with Subcomponents-level Attention","date":"2024-12-20","arxiv_id":"2412.15602","n_code_links":0,"syntology":null},{"paper":null,"title":"The Two-Hop Curse: LLMs trained on A$\\rightarrow$B, B$\\rightarrow$C fail to learn A$\\rightarrow$C","date":"2024-11-25","arxiv_id":"2411.16353","n_code_links":0,"syntology":null},{"paper":null,"title":"SOAR: Self-Occluded Avatar Recovery from a Single Video In the Wild","date":"2024-10-31","arxiv_id":"2410.23800","n_code_links":0,"syntology":null},{"paper":null,"title":"PerTok: Expressive Encoding and Modeling of Symbolic Musical Ideas and Variations","date":"2024-10-02","arxiv_id":"2410.02060","n_code_links":0,"syntology":null},{"paper":null,"title":"GLMHA A Guided Low-rank Multi-Head Self-Attention for Efficient Image Restoration and Spectral Reconstruction","date":"2024-10-01","arxiv_id":"2410.00380","n_code_links":0,"syntology":null},{"paper":"/paper/large-language-models-versus-classical","title":"Large Language Models versus Classical Machine Learning: Performance in COVID-19 Mortality Prediction Using High-Dimensional Tabular Data","date":"2024-09-02","arxiv_id":"2409.02136","n_code_links":2,"syntology":null},{"paper":null,"title":"Equitable Skin Disease Prediction Using Transfer Learning and Domain Adaptation","date":"2024-09-01","arxiv_id":"2409.00873","n_code_links":0,"syntology":null},{"paper":null,"title":"Tangram: Benchmark for Evaluating Geometric Element Recognition in Large Multimodal Models","date":"2024-08-25","arxiv_id":"2408.13854","n_code_links":0,"syntology":null},{"paper":"/paper/gabinsight-exploring-gender-activity-binding","title":"GABInsight: Exploring Gender-Activity Binding Bias in Vision-Language Models","date":"2024-07-30","arxiv_id":"2407.21001","n_code_links":1,"syntology":null},{"paper":"/paper/motamot-a-dataset-for-revealing-the-supremacy","title":"Motamot: A Dataset for Revealing the Supremacy of Large Language Models over Transformer Models in Bengali Political Sentiment Analysis","date":"2024-07-28","arxiv_id":"2407.19528","n_code_links":1,"syntology":null},{"paper":null,"title":"scGHSOM: Hierarchical clustering and visualization of single-cell and CRISPR data using growing hierarchical SOM","date":"2024-07-24","arxiv_id":"2407.16984","n_code_links":0,"syntology":null},{"paper":"/paper/predicting-the-best-of-n-visual-trackers","title":"Predicting the Best of N Visual Trackers","date":"2024-07-22","arxiv_id":"2407.15707","n_code_links":1,"syntology":null},{"paper":null,"title":"Lightweight Large Language Model for Medication Enquiry: Med-Pal","date":"2024-07-02","arxiv_id":"2407.12822","n_code_links":0,"syntology":null},{"paper":null,"title":"Deep Reinforcement Learning Strategies in Finance: Insights into Asset Holding, Trading Behavior, and Purchase Diversity","date":"2024-06-29","arxiv_id":"2407.09557","n_code_links":0,"syntology":null},{"paper":null,"title":"Modeling the Real World with High-Density Visual Particle Dynamics","date":"2024-06-28","arxiv_id":"2406.19800","n_code_links":0,"syntology":null},{"paper":"/paper/hopping-too-late-exploring-the-limitations-of","title":"Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries","date":"2024-06-18","arxiv_id":"2406.12775","n_code_links":1,"syntology":{"ran":10,"of":12,"unverified":2,"pointer_only":0}},{"paper":"/paper/smart-expert-system-large-language-models-as","title":"Adaptable and Reliable Text Classification using Large Language Models","date":"2024-05-17","arxiv_id":"2405.10523","n_code_links":1,"syntology":null},{"paper":null,"title":"Designing and Evaluating Dialogue LLMs for Co-Creative Improvised Theatre","date":"2024-05-11","arxiv_id":"2405.07111","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/language-modelling","name":"Language Modelling","papers":6},{"task":"/task/decoder","name":"Decoder","papers":5},{"task":"/task/time-series-1","name":"Time Series","papers":5},{"task":"/task/classification-1","name":"Classification","papers":4},{"task":"/task/language-modeling","name":"Language Modeling","papers":4},{"task":"/task/time-series","name":"Time Series Analysis","papers":4},{"task":"/task/anomaly-detection","name":"Anomaly Detection","papers":3},{"task":"/task/computational-efficiency","name":"Computational Efficiency","papers":3},{"task":"/task/decision-making","name":"Decision Making","papers":3},{"task":null,"name":"GPU","papers":3},{"task":"/task/image-classification","name":"Image Classification","papers":3},{"task":"/task/nerf","name":"NeRF","papers":3},{"task":"/task/novel-view-synthesis","name":"Novel View Synthesis","papers":3},{"task":"/task/segmentation","name":"Segmentation","papers":3},{"task":"/task/sentiment-analysis","name":"Sentiment Analysis","papers":3},{"task":"/task/image-classification","name":"image-classification","papers":3},{"task":"/task/autonomous-driving","name":"Autonomous Driving","papers":2},{"task":"/task/benchmarking","name":"Benchmarking","papers":2},{"task":"/task/clustering","name":"Clustering","papers":2},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":2}],"tasks_shown":20,"n_tasks":143,"usage_by_year":[{"year":"2020","papers":3},{"year":"2021","papers":17},{"year":"2022","papers":23},{"year":"2023","papers":23},{"year":"2024","papers":25},{"year":"2025","papers":11}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/favor"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}