Methods › General › Attention Mechanisms › FAVOR+
Fast Attention Via Positive Orthogonal Random Features
FAVOR+
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
FAVOR+, or Fast Attention Via Positive Orthogonal Random Features, is an efficient attention mechanism used in the Performer architecture which leverages approaches such as kernel methods and random features approximation for approximating softmax and Gaussian kernels.
FAVOR+ works for attention blocks using matrices A ∈ℝ^(L×L) of the form A(i, j) = K(qᵢᵀ, kⱼᵀ), with qᵢ/kⱼ standing for the iᵗʰ/jᵗʰ query/key row-vector in Q/K and kernel K : ℝᵈ × ℝᵈ →ℝ₊ defined for the (usually randomized) mapping: ϕ: ℝᵈ → ℝʳ₊ (for some r > 0) as:
K(x, y) = E[ϕ(x)ᵀϕ(y)]
We call ϕ(u) a random feature map for u ∈ℝᵈ . For Q^’, K^’ ∈ℝ^(L ×r) with rows given as ϕ(qᵢᵀ)ᵀ and ϕ(kᵢᵀ)ᵀ respectively, this leads directly to the efficient attention mechanism of the form:
Ât̂t̂_̂↔̂(Q, K, V) = D̂⁻¹(Q^’((K^’)ᵀV))
where
D̂ = diag(Q^’((K^’)1_L))
The above scheme constitutes the FA-part of the FAVOR+ mechanism. The other parts are achieved by:
- The R part : The softmax kernel is approximated though trigonometric functions, in the form of a regularized softmax-kernel SMREG, that employs positive random features (PRFs).
- The OR+ part : To reduce the variance of the estimator, so we can use a smaller number of random features, different samples are entangled to be exactly orthogonal using the Gram-Schmidt orthogonalization procedure.
The details are quite technical, so it is recommended you read the paper for further information on these steps.
In Syntology repository known, no verified implementation (declared_code; repositories: 5 listed of 7 in the graph, strongest link: official). The archive credits no paper with an arXiv id for this method, so whether this answer (about arXiv:2009.14794) is about this method is not checked.
Code Syntology holds under other names, as the lookup lists them (code_nearby); not a complete list, and none of them is this method.
- Performers · same_paper (from the same paper; a different method) · level V3 · retrieve with
get_reference_implementation(name='Performers')
Origin paper of this answer, arXiv:2009.14794 (its page on this site · its page on Syntology): Syntology's run record for its harvested code, 11 ran (of which 3 constructed an object rather than computing a result; 7 with no instrument failure: 1 honoured, 0 violated, 6 with no contract checked; 4 where Syntology's instrument failed) · 5 unverified (of 16 harvested samples) · 6 pointer-only (licence). The instrument figure counts failures of Syntology's instrument, not of the code. Ran means executed on a synthesized input, not that the code is correct.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
What each level means, quoted from Syntology's server card: V0 “imports”; V1 “executes against a synthesized fixture and returns a value of the recorded type”; V2 “two implementations GENERATED from the paper's signature and spec agree numerically (1e-6) on fixture inputs” “neither of the two ever read the authors' own code, so V2 measures how unambiguously the paper specifies the method, not fidelity to the authors' implementation (measured 2026-09-11, commit 75eaa77: cross_checked_by is a second LLM on 2,628 of 2,831 served samples)”; V3 “paper-derived property tests pass on both of those same two generations.” The card adds: “No rung on this ladder compares the served code against the authors' implementation, so treat every level as evidence about runnability and internal agreement, not about matching the paper's results.” A level quoted in two parts is one definition the card writes as two clauses joined by a double hyphen, each clause quoted on its own (how these levels are quoted).
Papers archive 2025-07-28
archive papers tagged: 102 · with a code link: 43 · where Syntology ran a sample: 9 (8 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (9 of 102 tagged: 8 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)
Page 1 of 2: papers 1 to 100 of 102, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists 2 Jun 2025 · 0 repositories · arXiv:2506.01241
-
WirelessMathBench: A Mathematical Modeling Benchmark for LLMs in Wireless Communications 20 May 2025 · 0 repositories · arXiv:2505.14354
-
CacheFormer: High Attention-Based Segment Caching 18 Apr 2025 · 0 repositories · arXiv:2504.13981
-
Deconstructing Jazz Piano Style Using Machine Learning 7 Apr 2025 · 0 repositories · arXiv:2504.05009
-
Predicting Survivability of Cancer Patients with Metastatic Patterns Using Explainable AI 7 Apr 2025 · 1 repository · arXiv:2504.06306
-
Forecasting Empty Container availability for Vehicle Booking System Application 14 Mar 2025 · 0 repositories · arXiv:2503.11728
-
STEAD: Spatio-Temporal Efficient Anomaly Detection for Time and Compute Sensitive Applications 11 Mar 2025 · 1 repository · arXiv:2503.07942
-
Deep Learning-Based Approach for Automatic 2D and 3D MRI Segmentation of Gliomas 27 Feb 2025 · 0 repositories · arXiv:2502.19760
-
On the use of Performer and Agent Attention for Spoken Language Identification 9 Feb 2025 · 0 repositories · arXiv:2502.05841
-
Nick Patrick Contreras 9 Jan 2025 · 0 repositories
-
Comparative Study of Deep Learning Architectures for Textual Damage Level Classification 3 Jan 2025 · 0 repositories · arXiv:2501.01694
-
ELECTRA and GPT-4o: Cost-Effective Partners for Sentiment Analysis 29 Dec 2024 · 1 repository · arXiv:2501.00062
-
Music Genre Classification: Ensemble Learning with Subcomponents-level Attention 20 Dec 2024 · 0 repositories · arXiv:2412.15602
-
The Two-Hop Curse: LLMs trained on A→B, B→C fail to learn A→C 25 Nov 2024 · 0 repositories · arXiv:2411.16353
-
SOAR: Self-Occluded Avatar Recovery from a Single Video In the Wild 31 Oct 2024 · 0 repositories · arXiv:2410.23800
-
PerTok: Expressive Encoding and Modeling of Symbolic Musical Ideas and Variations 2 Oct 2024 · 0 repositories · arXiv:2410.02060
-
GLMHA A Guided Low-rank Multi-Head Self-Attention for Efficient Image Restoration and Spectral Reconstruction 1 Oct 2024 · 0 repositories · arXiv:2410.00380
-
Large Language Models versus Classical Machine Learning: Performance in COVID-19 Mortality Prediction Using High-Dimensional Tabular Data 2 Sep 2024 · 2 repositories · arXiv:2409.02136
-
Equitable Skin Disease Prediction Using Transfer Learning and Domain Adaptation 1 Sep 2024 · 0 repositories · arXiv:2409.00873
-
Tangram: Benchmark for Evaluating Geometric Element Recognition in Large Multimodal Models 25 Aug 2024 · 0 repositories · arXiv:2408.13854
-
GABInsight: Exploring Gender-Activity Binding Bias in Vision-Language Models 30 Jul 2024 · 1 repository · arXiv:2407.21001
-
Motamot: A Dataset for Revealing the Supremacy of Large Language Models over Transformer Models in Bengali Political Sentiment Analysis 28 Jul 2024 · 1 repository · arXiv:2407.19528
-
scGHSOM: Hierarchical clustering and visualization of single-cell and CRISPR data using growing hierarchical SOM 24 Jul 2024 · 0 repositories · arXiv:2407.16984
-
Predicting the Best of N Visual Trackers 22 Jul 2024 · 1 repository · arXiv:2407.15707
-
Lightweight Large Language Model for Medication Enquiry: Med-Pal 2 Jul 2024 · 0 repositories · arXiv:2407.12822
-
Deep Reinforcement Learning Strategies in Finance: Insights into Asset Holding, Trading Behavior, and Purchase Diversity 29 Jun 2024 · 0 repositories · arXiv:2407.09557
-
Modeling the Real World with High-Density Visual Particle Dynamics 28 Jun 2024 · 0 repositories · arXiv:2406.19800
-
Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries 18 Jun 2024 · 1 repository · arXiv:2406.12775Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 5 where Syntology's instrument failed) · 2 unverified (of 12 harvested samples)
-
Adaptable and Reliable Text Classification using Large Language Models 17 May 2024 · 1 repository · arXiv:2405.10523
-
Designing and Evaluating Dialogue LLMs for Co-Creative Improvised Theatre 11 May 2024 · 0 repositories · arXiv:2405.07111
-
Analyzing Domestic Violence through Exploratory Data Analysis and Explainable Ensemble Learning Insights 22 Mar 2024 · 0 repositories · arXiv:2403.15594
-
SynerMix: Synergistic Mixup Solution for Enhanced Intra-Class Cohesion and Inter-Class Separability in Image Classification 21 Mar 2024 · 2 repositories · arXiv:2403.14137
-
Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Prompts 11 Mar 2024 · 2 repositories · arXiv:2403.06495Syntology official (archive's flag): 16 ran · 16 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 1 violated, 8 with no contract checked; 7 where Syntology's instrument failed) · 5 unverified (of 21 harvested samples) · 16 pointer-only (licence)
-
Enhancing Price Prediction in Cryptocurrency Using Transformer Neural Network and Technical Indicators 6 Mar 2024 · 0 repositories · arXiv:2403.03606
-
A Change Detection Reality Check 10 Feb 2024 · 1 repository · arXiv:2402.06994
-
DiffPerformer: Iterative Learning of Consistent Latent Guidance for Diffusion-based Human Video Generation 1 Jan 2024 · 0 repositories
-
Hate Speech and Offensive Content Detection in Indo-Aryan Languages: A Battle of LSTM and Transformers 9 Dec 2023 · 0 repositories · arXiv:2312.05671
-
GauHuman: Articulated Gaussian Splatting from Monocular Human Videos 5 Dec 2023 · 1 repository · arXiv:2312.02973Syntology official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 3 honoured, 0 violated, 1 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified (of 10 harvested samples) · 10 pointer-only (licence)
-
Comprehensive framework for evaluation of deep neural networks in detection and quantification of lymphoma from PET/CT images: clinical insights, pitfalls, and observer agreement analyses 16 Nov 2023 · 1 repository · arXiv:2311.09614
-
An Intelligent Social Learning-based Optimization Strategy for Black-box Robotic Control with Reinforcement Learning 11 Nov 2023 · 0 repositories · arXiv:2311.06576
-
GeoFormer: Predicting Human Mobility using Generative Pre-trained Transformer (GPT) 9 Nov 2023 · 0 repositories · arXiv:2311.05092
-
Inside the black box: Neural network-based real-time prediction of US recessions 26 Oct 2023 · 0 repositories · arXiv:2310.17571
-
Systematic Assessment of Factual Knowledge in Large Language Models 18 Oct 2023 · 0 repositories · arXiv:2310.11638
-
Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion 5 Oct 2023 · 1 repository · arXiv:2310.03502Syntology official (archive's flag): 3 ran · 12 ran (of which 1 constructed an object rather than computing a result; 5 with no instrument failure: 3 honoured, 0 violated, 2 with no contract checked; 7 where Syntology's instrument failed) · 6 unverified (of 18 harvested samples) · 10 pointer-only (licence)
-
PolySketchFormer: Fast Transformers via Sketching Polynomial Kernels 2 Oct 2023 · 0 repositories · arXiv:2310.01655
-
HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World 29 Sep 2023 · 0 repositories · arXiv:2309.17024
-
Nominality Score Conditioned Time Series Anomaly Detection by Point/Sequential Reconstruction 21 Sep 2023 · 1 repository
-
Large-scale Weakly Supervised Learning for Road Extraction from Satellite Imagery 14 Sep 2023 · 0 repositories · arXiv:2309.07823
-
Stringesthesia: Dynamically Shifting Musical Agency Between Audience and Performer Based on Trust in an Interactive and Improvised Performance 11 Sep 2023 · 0 repositories · arXiv:2309.05786
-
DECODE: Data-driven Energy Consumption Prediction leveraging Historical Data and Environmental Factors in Buildings 6 Sep 2023 · 0 repositories · arXiv:2309.02908
-
GHuNeRF: Generalizable Human NeRF from a Monocular Video 31 Aug 2023 · 1 repository · arXiv:2308.16576Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
JAZZVAR: A Dataset of Variations found within Solo Piano Performances of Jazz Standards for Music Overpainting 18 Jul 2023 · 1 repository · arXiv:2307.09670
-
Sumformer: Universal Approximation for Efficient Transformers 5 Jul 2023 · 0 repositories · arXiv:2307.02301
-
Using Sequences of Life-events to Predict Human Lives 5 Jun 2023 · 2 repositories · arXiv:2306.03009
-
R2H: Building Multimodal Navigation Helpers that Respond to Help Requests 23 May 2023 · 0 repositories · arXiv:2305.14260
-
Deep Learning Methods for Extracting Metaphorical Names of Flowers and Plants 18 May 2023 · 0 repositories · arXiv:2305.10833
-
TSI-GAN: Unsupervised Time Series Anomaly Detection using Convolutional Cycle-Consistent Generative Adversarial Networks 22 Mar 2023 · 1 repository · arXiv:2303.12952
-
Towards Unbounded Machine Unlearning 20 Feb 2023 · 1 repository · arXiv:2302.09880
-
Automatic Modulation Classification with Deep Neural Networks 27 Jan 2023 · 1 repository · arXiv:2301.11773
-
Robust representations of oil wells' intervals via sparse attention mechanism 29 Dec 2022 · 2 repositories · arXiv:2212.14246
-
Gaussian Process Classification Bandits 26 Dec 2022 · 0 repositories · arXiv:2212.13157
-
Human Performance Modeling and Rendering via Neural Animated Mesh 18 Sep 2022 · 1 repository · arXiv:2209.08468
-
Multiple View Performers for Shape Completion 13 Sep 2022 · 0 repositories · arXiv:2209.06291
-
Model-Free Deep Reinforcement Learning in Software-Defined Networks 3 Sep 2022 · 0 repositories · arXiv:2209.01490
-
NRBdMF: A recommendation algorithm for predicting drug effects considering directionality 5 Aug 2022 · 1 repository · arXiv:2208.04312
-
Drivable Volumetric Avatars using Texel-Aligned Features 20 Jul 2022 · 0 repositories · arXiv:2207.09774
-
NDF: Neural Deformable Fields for Dynamic Human Modelling 19 Jul 2022 · 1 repository · arXiv:2207.09193
-
Memory-free Online Change-point Detection: A Novel Neural Network Approach 8 Jul 2022 · 1 repository · arXiv:2207.03932
-
Machine and Deep Learning Applications to Mouse Dynamics for Continuous User Authentication 26 May 2022 · 1 repository · arXiv:2205.13646
-
Generalizable Neural Performer: Learning Robust Radiance Fields for Human Novel View Synthesis 25 Apr 2022 · 1 repository · arXiv:2204.11798Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples) · 4 pointer-only (licence)
-
Performer: A Novel PPG-to-ECG Reconstruction Transformer for a Digital Biomarker of Cardiovascular Disease Detection 25 Apr 2022 · 0 repositories · arXiv:2204.11795
-
On Variants of Root Normalised Order-aware Divergence and a Divergence based on Kendall's Tau 15 Apr 2022 · 0 repositories · arXiv:2204.07304
-
Neural Rendering of Humans in Novel View and Pose from Monocular Video 4 Apr 2022 · 0 repositories · arXiv:2204.01218
-
Acoustics-specific Piano Velocity Estimation 30 Mar 2022 · 1 repository · arXiv:2203.16294
-
UV Volumes for Real-time Rendering of Editable Free-view Human Performance 27 Mar 2022 · 1 repository · arXiv:2203.14402Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 5 harvested samples) · 1 pointer-only (licence)
-
Prioritized Variable-length Test Cases Generation for Finite State Machines 17 Mar 2022 · 0 repositories · arXiv:2203.09596
-
The NLP Task Effectiveness of Long-Range Transformers 16 Feb 2022 · 0 repositories · arXiv:2202.07856
-
Deep Performer: Score-to-Audio Music Performance Synthesis 12 Feb 2022 · 0 repositories · arXiv:2202.06034
-
Automotive Parts Assessment: Applying Real-time Instance-Segmentation Models to Identify Vehicle Parts 2 Feb 2022 · 0 repositories · arXiv:2202.00884
-
Convolutional Xformers for Vision 25 Jan 2022 · 1 repository · arXiv:2201.10271
-
Looking at the Performer from a Hopfield Point of View 17 Jan 2022 · 0 repositories
-
Enhancement of Healthcare Data Performance Metrics using Neural Network Machine Learning Algorithms 16 Jan 2022 · 0 repositories · arXiv:2201.05962
-
Efficient Data Compression for 3D Sparse TPC via Bicephalous Convolutional Autoencoder 9 Nov 2021 · 1 repository · arXiv:2111.05423
-
Losses, Dissonances, and Distortions 8 Nov 2021 · 0 repositories · arXiv:2111.05128
-
PhyloTransformer: A Discriminative Model for Mutation Prediction Based on a Multi-head Self-attention Mechanism 3 Nov 2021 · 0 repositories · arXiv:2111.01969
-
Transformers Generalize DeepSets and Can be Extended to Graphs and Hypergraphs 27 Oct 2021 · 2 repositories · arXiv:2110.14416Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
Pyramidal Blur Aware X-Corner Chessboard Detector 26 Oct 2021 · 0 repositories · arXiv:2110.13793
-
Multimodal Performers for Genomic Selection and Crop Yield Prediction 20 Oct 2021 · 1 repository
-
Audiomer: A Convolutional Transformer For Keyword Spotting 21 Sep 2021 · 0 repositories · arXiv:2109.10252
-
Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering 15 Sep 2021 · 1 repository · arXiv:2109.07448
-
PermuteFormer: Efficient Relative Position Encoding for Long Sequences 6 Sep 2021 · 1 repository · arXiv:2109.02377Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (of 1 harvested sample) · 1 pointer-only (licence)
-
SOMTimeS: Self Organizing Maps for Time Series Clustering and its Application to Serious Illness Conversations 26 Aug 2021 · 0 repositories · arXiv:2108.11523
-
Performer Identification From Symbolic Representation of Music Using Statistical Models 5 Aug 2021 · 0 repositories · arXiv:2108.02576
-
Neural Relighting and Expression Transfer On Video Portraits 30 Jul 2021 · 0 repositories · arXiv:2107.14735
-
Vision Xformers: Efficient Attention for Image Classification 5 Jul 2021 · 2 repositories · arXiv:2107.02239
-
Memory-efficient Transformers via Top-k Attention 13 Jun 2021 · 2 repositories · arXiv:2106.06899
-
Sequential Online Chore Division for Autonomous Vehicle Convoy Formation 9 Apr 2021 · 0 repositories · arXiv:2104.04159
-
A Practical Survey on Faster and Lighter Transformers 26 Mar 2021 · 0 repositories · arXiv:2103.14636
-
The FaCells. An Exploratory Study about LSTM Layers on Face Sketches Classifiers 22 Feb 2021 · 1 repository · arXiv:2102.11361
-
Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans 31 Dec 2020 · 3 repositories · arXiv:2012.15838Syntology community repositories only · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
Tasks archive 2025-07-28
20 shown of 143 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections