{"url":"/method/global-local-attention","slug":"global-local-attention","name":"Global-Local Attention","full_name":"Global-Local Attention","full_name_withheld":false,"description_markdown":"**Global-Local Attention** is a type of attention mechanism used in the [ETC](https://paperswithcode.com/method/etc) architecture. ETC receives two separate input sequences: the global input $x^{g} = (x^{g}\\_{1}, \\dots, x^{g}\\_{n\\_{g}})$ and the long input $x^{l} = (x^{l}\\_{1}, \\dots x^{l}\\_{n\\_{l}})$. Typically, the long input contains the input a [standard Transformer](https://paperswithcode.com/method/transformer) would receive, while the global input contains a much smaller number of auxiliary tokens ($n\\_{g}  \\ll n\\_{l}$). Attention is then split into four separate pieces: global-to-global (g2g), global-tolong (g2l), long-to-global (l2g), and long-to-long (l2l). Attention in the l2l piece (the most computationally expensive piece) is restricted to a fixed radius $r \\ll n\\_{l}$. To compensate for this limited attention span, the tokens in the global input have unrestricted attention, and thus long input tokens can transfer information to each other through global input tokens. Accordingly, g2g, g2l, and l2g pieces of attention are unrestricted.","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"https://arxiv.org/abs/2004.08483v5","title":"ETC: Encoding Long and Structured Inputs in Transformers","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Attention Mechanisms","url":"/methods/category/attention-mechanisms","pwc_aliases":["attention-mechanisms-1"]}],"n_papers_tagged":56,"archive_num_papers":null,"papers_newest_first":[{"paper":null,"title":"Koopman-Based Event-Triggered Control from Data","date":"2025-04-19","arxiv_id":"2504.14334","n_code_links":0,"syntology":null},{"paper":null,"title":"Integrating Locality-Aware Attention with Transformers for General Geometry PDEs","date":"2025-04-18","arxiv_id":"2504.13480","n_code_links":0,"syntology":null},{"paper":null,"title":"Imitating Radiological Scrolling: A Global-Local Attention Model for 3D Chest CT Volumes Multi-Label Anomaly Classification","date":"2025-03-26","arxiv_id":"2503.20652","n_code_links":0,"syntology":null},{"paper":"/paper/a-diffusion-model-and-knowledge-distillation","title":"A Diffusion Model and Knowledge Distillation Framework for Robust Coral Detection in Complex Underwater Environments","date":"2025-01-06","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"title":"Performance-Barrier Event-Triggered PDE Control of Traffic Flow","date":"2025-01-01","arxiv_id":"2501.00722","n_code_links":0,"syntology":null},{"paper":null,"title":"Automated Toll Management System Using RFID and Image Processing","date":"2024-12-02","arxiv_id":"2412.01728","n_code_links":0,"syntology":null},{"paper":null,"title":"DELTA: Dense Efficient Long-range 3D Tracking for any video","date":"2024-10-31","arxiv_id":"2410.24211","n_code_links":0,"syntology":null},{"paper":null,"title":"Hierarchical Event-Triggered Systems: Safe Learning of Quasi-Optimal Deadline Policies","date":"2024-09-15","arxiv_id":"2409.09812","n_code_links":0,"syntology":null},{"paper":null,"title":"MicroEmo: Time-Sensitive Multimodal Emotion Recognition with Micro-Expression Dynamics in Video Dialogues","date":"2024-07-23","arxiv_id":"2407.16552","n_code_links":0,"syntology":null},{"paper":null,"title":"Performance-Barrier Event-Triggered Control of a Class of Reaction-Diffusion PDEs","date":"2024-07-11","arxiv_id":"2407.08178","n_code_links":0,"syntology":null},{"paper":null,"title":"A Global-Local Attention Mechanism for Relation Classification","date":"2024-07-01","arxiv_id":"2407.01424","n_code_links":0,"syntology":null},{"paper":"/paper/sckansformer-fine-grained-classification-of","title":"SCKansformer: Fine-Grained Classification of Bone Marrow Cells via Kansformer Backbone and Hierarchical Attention Mechanisms","date":"2024-06-14","arxiv_id":"2406.09931","n_code_links":1,"syntology":null},{"paper":null,"title":"Contextual Dynamic Pricing: Algorithms, Optimality, and Local Differential Privacy Constraints","date":"2024-06-04","arxiv_id":"2406.02424","n_code_links":0,"syntology":null},{"paper":"/paper/a-challenge-dataset-and-effective-models-for-1","title":"A Challenge Dataset and Effective Models for Conversational Stance Detection","date":"2024-03-17","arxiv_id":"2403.11145","n_code_links":1,"syntology":null},{"paper":null,"title":"Event-Triggered Robust Cooperative Output Regulation for a Class of Linear Multi-Agent Systems with an Unknown Exosystem","date":"2024-03-01","arxiv_id":"2403.00645","n_code_links":0,"syntology":null},{"paper":"/paper/griffin-mixing-gated-linear-recurrences-with","title":"Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models","date":"2024-02-29","arxiv_id":"2402.19427","n_code_links":4,"syntology":{"ran":0,"of":10,"unverified":10,"pointer_only":0}},{"paper":"/paper/inversematrixvt3d-an-efficient-projection","title":"InverseMatrixVT3D: An Efficient Projection Matrix-Based Approach for 3D Occupancy Prediction","date":"2024-01-23","arxiv_id":"2401.12422","n_code_links":1,"syntology":null},{"paper":"/paper/dgl-dynamic-global-local-prompt-tuning-for","title":"DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval","date":"2024-01-19","arxiv_id":"2401.10588","n_code_links":2,"syntology":{"ran":2,"of":2,"unverified":0,"pointer_only":1}},{"paper":null,"title":"Replication-proof Bandit Mechanism Design with Bayesian Agents","date":"2023-12-28","arxiv_id":"2312.16896","n_code_links":0,"syntology":null},{"paper":null,"title":"Learning-based Scheduling for Information Accuracy and Freshness in Wireless Networks","date":"2023-10-24","arxiv_id":"2310.15705","n_code_links":0,"syntology":null},{"paper":"/paper/deformux-net-exploring-a-3d-foundation","title":"DeformUX-Net: Exploring a 3D Foundation Backbone for Medical Image Segmentation with Depthwise Deformable Convolution","date":"2023-09-30","arxiv_id":"2310.00199","n_code_links":1,"syntology":null},{"paper":null,"title":"Listen to Minority: Encrypted Traffic Classification for Class Imbalance with Contrastive Pre-Training","date":"2023-08-31","arxiv_id":"2308.16453","n_code_links":0,"syntology":null},{"paper":"/paper/integrating-a-heterogeneous-graph-with-entity","title":"Integrating a Heterogeneous Graph with Entity-aware Self-attention using Relative Position Labels for Reading Comprehension Model","date":"2023-07-19","arxiv_id":"2307.10443","n_code_links":0,"syntology":null},{"paper":null,"title":"High-dimensional Contextual Bandit Problem without Sparsity","date":"2023-06-19","arxiv_id":"2306.11017","n_code_links":0,"syntology":null},{"paper":null,"title":"Permutation Decision Trees","date":"2023-06-05","arxiv_id":"2306.02617","n_code_links":0,"syntology":null},{"paper":null,"title":"An Improved Heart Disease Prediction Using Stacked Ensemble Method","date":"2023-04-12","arxiv_id":"2304.06015","n_code_links":0,"syntology":null},{"paper":null,"title":"Asynchronous Event-Triggered Control for Non-Linear Systems","date":"2022-11-25","arxiv_id":"2211.13846","n_code_links":0,"syntology":null},{"paper":null,"title":"LittleBird: Efficient Faster & Longer Transformer for Question Answering","date":"2022-10-21","arxiv_id":"2210.11870","n_code_links":0,"syntology":null},{"paper":null,"title":"Short Text Pre-training with Extended Token Classification for E-commerce Query Understanding","date":"2022-10-08","arxiv_id":"2210.03915","n_code_links":0,"syntology":null},{"paper":null,"title":"Embed to Control Partially Observed Systems: Representation Learning with Provable Sample Efficiency","date":"2022-05-26","arxiv_id":"2205.13476","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/classification-1","name":"Classification","papers":3},{"task":"/task/language-modelling","name":"Language Modelling","papers":3},{"task":"/task/object-detection","name":"Object Detection","papers":3},{"task":"/task/scheduling","name":"Scheduling","papers":3},{"task":"/task/object-detection-1","name":"object-detection","papers":3},{"task":"/task/computational-efficiency","name":"Computational Efficiency","papers":2},{"task":"/task/emotion-recognition","name":"Emotion Recognition","papers":2},{"task":null,"name":"GPU","papers":2},{"task":"/task/mamba","name":"Mamba","papers":2},{"task":"/task/management","name":"Management","papers":2},{"task":"/task/multi-armed-bandits","name":"Multi-Armed Bandits","papers":2},{"task":null,"name":"Position","papers":2},{"task":"/task/question-answering","name":"Question Answering","papers":2},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":2},{"task":"/task/representation-learning","name":"Representation Learning","papers":2},{"task":"/task/retrieval","name":"Retrieval","papers":2},{"task":"/task/scene-segmentation","name":"Scene Segmentation","papers":2},{"task":"/task/segmentation","name":"Segmentation","papers":2},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":2},{"task":"/task/thompson-sampling","name":"Thompson Sampling","papers":2}],"tasks_shown":20,"n_tasks":92,"usage_by_year":[{"year":"2020","papers":8},{"year":"2021","papers":13},{"year":"2022","papers":9},{"year":"2023","papers":8},{"year":"2024","papers":13},{"year":"2025","papers":5}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/global-local-attention"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}