{"url":"/method/lsh-attention","slug":"lsh-attention","name":"LSH Attention","full_name":"Locality Sensitive Hashing Attention","full_name_withheld":false,"description_markdown":"**LSH Attention**, or **Locality Sensitive Hashing Attention** is a replacement for [dot-product attention](https://paperswithcode.com/method/scaled) with one that uses locality-sensitive hashing, changing its complexity from O($L^2$) to O($L\\log L$), where $L$ is the length of the sequence. LSH refers to a family of functions (known as LSH families) to hash data points into buckets so that data points near each other are located in the same buckets with high probability, while data points far from each other are likely to be in different buckets. It was proposed as part of the [Reformer](https://paperswithcode.com/method/reformer) architecture.","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"https://arxiv.org/abs/2001.04451v2","title":"Reformer: The Efficient Transformer","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/google/trax/blob/af3a38917bd1bc69cf5d25ce007e16185f22f050/trax/layers/research/efficient_attention.py#L1084","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Attention Mechanisms","url":"/methods/category/attention-mechanisms","pwc_aliases":["attention-mechanisms-1"]}],"n_papers_tagged":22,"archive_num_papers":null,"papers_newest_first":[{"paper":"/paper/cardioformer-advancing-ai-in-ecg-analysis","title":"Cardioformer: Advancing AI in ECG Analysis with Multi-Granularity Patching and ResNet","date":"2025-05-08","arxiv_id":"2505.05538","n_code_links":1,"syntology":null},{"paper":null,"title":"Fast-Powerformer: A Memory-Efficient Transformer for Accurate Mid-Term Wind Power Forecasting","date":"2025-04-15","arxiv_id":"2504.10923","n_code_links":0,"syntology":null},{"paper":null,"title":"ReFormer: Generating Radio Fakes for Data Augmentation","date":"2024-12-31","arxiv_id":"2501.00282","n_code_links":0,"syntology":null},{"paper":null,"title":"GLMHA A Guided Low-rank Multi-Head Self-Attention for Efficient Image Restoration and Spectral Reconstruction","date":"2024-10-01","arxiv_id":"2410.00380","n_code_links":0,"syntology":null},{"paper":null,"title":"Masked Face Recognition with Generative-to-Discriminative Representations","date":"2024-05-27","arxiv_id":"2405.16761","n_code_links":0,"syntology":null},{"paper":null,"title":"A novel transformer-based approach for soil temperature prediction","date":"2023-11-20","arxiv_id":"2311.11626","n_code_links":0,"syntology":null},{"paper":null,"title":"LegalRelectra: Mixed-domain Language Modeling for Long-range Legal Text Comprehension","date":"2022-12-16","arxiv_id":"2212.08204","n_code_links":0,"syntology":null},{"paper":"/paper/fecam-frequency-enhanced-channel-attention","title":"FECAM: Frequency Enhanced Channel Attention Mechanism for Time Series Forecasting","date":"2022-12-02","arxiv_id":"2212.01209","n_code_links":1,"syntology":null},{"paper":"/paper/non-stationary-transformers-rethinking-the","title":"Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting","date":"2022-05-28","arxiv_id":"2205.14415","n_code_links":2,"syntology":null},{"paper":null,"title":"Discovering material information using hierarchical Reformer model on financial regulatory filings","date":"2022-03-28","arxiv_id":"2204.05979","n_code_links":0,"syntology":null},{"paper":"/paper/dynamic-token-normalization-improves-vision-1","title":"Dynamic Token Normalization Improves Vision Transformers","date":"2021-12-05","arxiv_id":"2112.02624","n_code_links":1,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":1}},{"paper":"/paper/hierarchical-transformers-are-more-efficient","title":"Hierarchical Transformers Are More Efficient Language Models","date":"2021-10-26","arxiv_id":"2110.13711","n_code_links":3,"syntology":{"ran":2,"of":4,"unverified":2,"pointer_only":2}},{"paper":"/paper/end-to-end-user-behavior-retrieval-in-click","title":"End-to-End User Behavior Retrieval in Click-Through RatePrediction Model","date":"2021-08-10","arxiv_id":"2108.04468","n_code_links":1,"syntology":null},{"paper":"/paper/reformer-the-relational-transformer-for-image","title":"ReFormer: The Relational Transformer for Image Captioning","date":"2021-07-29","arxiv_id":"2107.14178","n_code_links":1,"syntology":null},{"paper":null,"title":"A Practical Survey on Faster and Lighter Transformers","date":"2021-03-26","arxiv_id":"2103.14636","n_code_links":0,"syntology":null},{"paper":null,"title":"ReAssert: Deep Learning for Assert Generation","date":"2020-11-19","arxiv_id":"2011.09784","n_code_links":0,"syntology":null},{"paper":null,"title":"Learning to Unknot","date":"2020-10-28","arxiv_id":"2010.16263","n_code_links":0,"syntology":null},{"paper":null,"title":"Robustification of Segmentation Models Against Adversarial Perturbations In Medical Imaging","date":"2020-09-23","arxiv_id":"2009.11090","n_code_links":0,"syntology":null},{"paper":null,"title":"Efficient Transformers: A Survey","date":"2020-09-14","arxiv_id":"2009.06732","n_code_links":0,"syntology":null},{"paper":"/paper/segmenting-scientific-abstracts-into","title":"Segmenting Scientific Abstracts into Discourse Categories: A Deep Learning-Based Approach for Sparse Labeled Data","date":"2020-05-11","arxiv_id":"2005.05414","n_code_links":1,"syntology":null},{"paper":"/paper/neural-machine-translation-with-joint","title":"Neural Machine Translation with Joint Representation","date":"2020-02-16","arxiv_id":"2002.06546","n_code_links":1,"syntology":null},{"paper":"/paper/reformer-the-efficient-transformer-1","title":"Reformer: The Efficient Transformer","date":"2020-01-13","arxiv_id":"2001.04451","n_code_links":10,"syntology":{"ran":6,"of":8,"unverified":2,"pointer_only":0}}],"papers_shown":22,"tasks":[{"task":"/task/language-modelling","name":"Language Modelling","papers":3},{"task":"/task/deep-learning","name":"Deep Learning","papers":2},{"task":"/task/image-generation","name":"Image Generation","papers":2},{"task":"/task/language-modeling","name":"Language Modeling","papers":2},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":2},{"task":"/task/segmentation","name":"Segmentation","papers":2},{"task":"/task/sentence","name":"Sentence","papers":2},{"task":"/task/survey","name":"Survey","papers":2},{"task":"/task/time-series-1","name":"Time Series","papers":2},{"task":"/task/time-series","name":"Time Series Analysis","papers":2},{"task":"/task/time-series-forecasting","name":"Time Series Forecasting","papers":2},{"task":"/task/attribute","name":"Attribute","papers":1},{"task":"/task/machine-learning","name":"BIG-bench Machine Learning","papers":1},{"task":"/task/binary-classification","name":"Binary Classification","papers":1},{"task":"/task/click-through-rate-prediction","name":"Click-Through Rate Prediction","papers":1},{"task":"/task/computational-efficiency","name":"Computational Efficiency","papers":1},{"task":"/task/d4rl","name":"D4RL","papers":1},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":1},{"task":"/task/deblurring","name":"Deblurring","papers":1},{"task":"/task/decoder","name":"Decoder","papers":1}],"tasks_shown":20,"n_tasks":50,"usage_by_year":[{"year":"2020","papers":7},{"year":"2021","papers":5},{"year":"2022","papers":4},{"year":"2023","papers":1},{"year":"2024","papers":3},{"year":"2025","papers":2}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/lsh-attention"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}