{"url":"/method/non-local-operation","slug":"non-local-operation","name":"Non-Local Operation","full_name":"Non-Local Operation","full_name_withheld":false,"description_markdown":"A **Non-Local Operation** is a component for capturing long-range dependencies with deep neural networks. It is a generalization of the classical non-local mean operation in computer vision. Intuitively a non-local operation computes the response at a position as a weighted sum of the features at all positions in the input feature maps. The set of positions can be in space, time, or spacetime, implying that these operations are applicable for image, sequence, and video problems.\r\n\r\nFollowing the non-local mean operation, a generic non-local operation for deep neural networks is defined as:\r\n\r\n$$ \\mathbb{y}\\_{i} = \\frac{1}{\\mathcal{C}\\left(\\mathbb{x}\\right)}\\sum\\_{\\forall{j}}f\\left(\\mathbb{x}\\_{i}, \\mathbb{x}\\_{j}\\right)g\\left(\\mathbb{x}\\_{j}\\right) $$\r\n\r\nHere $i$ is the index of an output position (in space, time, or spacetime) whose response is to be computed and $j$ is the index that enumerates all possible positions. x is the input signal (image, sequence, video; often their features) and $y$ is the output signal of the same size as $x$. A pairwise function $f$ computes a scalar (representing relationship such as affinity) between $i$ and all $j$. The unary function $g$ computes a representation of the input signal at the position $j$. The\r\nresponse is normalized by a factor $C\\left(x\\right)$.\r\n\r\nThe non-local behavior is due to the fact that all positions ($\\forall{j}$) are considered in the operation. As a comparison, a convolutional operation sums up the weighted input in a local neighborhood (e.g., $i − 1 \\leq j \\leq i + 1$ in a 1D case with kernel size 3), and a recurrent operation at time $i$ is often based only on the current and the latest time steps (e.g., $j = i$ or $i − 1$).\r\n\r\nThe non-local operation is also different from a fully-connected (fc) layer. The equation above computes responses based on relationships between different locations, whereas fc uses learned weights. In other words, the relationship between $x\\_{j}$ and $x\\_{i}$ is not a function of the input data in fc, unlike in nonlocal layers. Furthermore, the formulation in the equation above supports inputs of variable sizes, and maintains the corresponding size in the output. On the contrary, an fc layer requires a fixed-size input/output and loses positional correspondence (e.g., that from $x\\_{i}$ to $y\\_{i}$ at the position $i$).\r\n\r\nA non-local operation is a flexible building block and can be easily used together with convolutional/recurrent layers. It can be added into the earlier part of deep neural networks, unlike fc layers that are often used in the end. This allows us to build a richer hierarchy that combines both non-local and local information.\r\n\r\nIn terms of parameterisation, we usually parameterise $g$ as a linear embedding of the form $g\\left(x\\_{j}\\right) = W\\_{g}\\mathbb{x}\\_{j}$ , where $W\\_{g}$ is a weight matrix to be learned. This is implemented as, e.g., 1×1 [convolution](https://paperswithcode.com/method/convolution) in space or 1×1×1 convolution in spacetime. For $f$ we use an affinity function, a list of which can be found [here](https://paperswithcode.com/methods/category/affinity-functions).","description_state":"present","introduced_year":null,"introduced_by":{"title":"Non-local Neural Networks","paper":"/paper/non-local-neural-networks","first_author":"Xiaolong Wang","n_authors":4,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/non-local-neural-networks"},"source":{"url":"http://arxiv.org/abs/1711.07971v3","title":"Non-local Neural Networks","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/tea1528/Non-Local-NN-Pytorch/blob/986937674eb3b85d3d3fbaaa8f384c0a26624121/models/non_local.py#L124","code_snippet_url_on_a_code_host":true,"categories":[{"area":"Computer Vision","area_id":"computer-vision","collection":"Image Feature Extractors","url":"/methods/category/image-feature-extractors","pwc_aliases":[]}],"n_papers_tagged":181,"archive_num_papers":181,"papers_newest_first":[{"paper":null,"title":"Robust Lane Detection with Wavelet-Enhanced Context Modeling and Adaptive Sampling","date":"2025-03-24","arxiv_id":"2503.18631","n_code_links":0,"syntology":null},{"paper":null,"title":"ParaGAN: A Scalable Distributed Training Framework for Generative Adversarial Networks","date":"2024-11-06","arxiv_id":"2411.03999","n_code_links":0,"syntology":null},{"paper":"/paper/unsupervised-panoptic-interpretation-of","title":"Unsupervised Panoptic Interpretation of Latent Spaces in GANs Using Space-Filling Vector Quantization","date":"2024-10-27","arxiv_id":"2410.20573","n_code_links":1,"syntology":null},{"paper":null,"title":"Enhancing Tree Type Detection in Forest Fire Risk Assessment: Multi-Stage Approach and Color Encoding with Forest Fire Risk Evaluation Framework for UAV Imagery","date":"2024-07-27","arxiv_id":"2407.19184","n_code_links":0,"syntology":null},{"paper":"/paper/a-scalable-quantum-non-local-neural-network","title":"A Scalable Quantum Non-local Neural Network for Image Classification","date":"2024-07-26","arxiv_id":"2407.18906","n_code_links":1,"syntology":null},{"paper":"/paper/unke-unstructured-knowledge-editing-in-large","title":"Everything is Editable: Extend Knowledge Editing to Unstructured Data in Large Language Models","date":"2024-05-24","arxiv_id":"2405.15349","n_code_links":1,"syntology":{"ran":5,"of":12,"unverified":7,"pointer_only":12}},{"paper":null,"title":"Deep Learning-Based CSI Feedback for XL-MIMO Systems in the Near-Field Domain","date":"2024-05-15","arxiv_id":"2405.09053","n_code_links":0,"syntology":null},{"paper":"/paper/vision-based-food-nutrition-estimation-via","title":"Vision-based Food Nutrition Estimation via RGB-D Fusion Network","date":"2023-10-25","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"title":"Accurate and lightweight dehazing via multi-receptive-field non-local network and novel contrastive regularization","date":"2023-09-28","arxiv_id":"2309.16494","n_code_links":0,"syntology":null},{"paper":null,"title":"On quantifying and improving realism of images generated with diffusion","date":"2023-09-26","arxiv_id":"2309.14756","n_code_links":0,"syntology":null},{"paper":null,"title":"Precision-Recall Divergence Optimization for Generative Modeling with GANs and Normalizing Flows","date":"2023-09-21","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/a-strategic-framework-for-optimal-decisions","title":"A Strategic Framework for Optimal Decisions in Football 1-vs-1 Shot-Taking Situations: An Integrated Approach of Machine Learning, Theory-Based Modeling, and Game Theory","date":"2023-07-27","arxiv_id":"2307.14732","n_code_links":2,"syntology":null},{"paper":"/paper/pyrus-base-an-open-source-python-framework","title":"Pyrus Base: An Open Source Python Framework for the RoboCup 2D Soccer Simulation","date":"2023-07-22","arxiv_id":"2307.16875","n_code_links":1,"syntology":null},{"paper":"/paper/diffusion-models-beat-gans-on-image","title":"Diffusion Models Beat GANs on Image Classification","date":"2023-07-17","arxiv_id":"2307.08702","n_code_links":1,"syntology":null},{"paper":null,"title":"Diversity is Strength: Mastering Football Full Game with Interactive Reinforcement Learning of Multiple AIs","date":"2023-06-28","arxiv_id":"2306.15903","n_code_links":0,"syntology":null},{"paper":null,"title":"Rosetta Neurons: Mining the Common Units in a Model Zoo","date":"2023-06-15","arxiv_id":"2306.09346","n_code_links":0,"syntology":null},{"paper":null,"title":"Toward more accurate and generalizable brain deformation estimators for traumatic brain injury detection with unsupervised domain adaptation","date":"2023-06-08","arxiv_id":"2306.05255","n_code_links":0,"syntology":null},{"paper":null,"title":"FOOCTTS: Generating Arabic Speech with Acoustic Environment for Football Commentator","date":"2023-06-07","arxiv_id":"2306.07936","n_code_links":0,"syntology":null},{"paper":null,"title":"Action valuation of on- and off-ball soccer players based on multi-agent deep reinforcement learning","date":"2023-05-29","arxiv_id":"2305.17886","n_code_links":0,"syntology":null},{"paper":"/paper/is-centralized-training-with-decentralized","title":"Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?","date":"2023-05-27","arxiv_id":"2305.17352","n_code_links":1,"syntology":{"ran":0,"of":1,"unverified":1,"pointer_only":1}},{"paper":null,"title":"Adaptive action supervision in reinforcement learning from real-world multi-agent demonstrations","date":"2023-05-22","arxiv_id":"2305.13030","n_code_links":0,"syntology":null},{"paper":"/paper/an-empirical-study-on-google-research","title":"An Empirical Study on Google Research Football Multi-agent Scenarios","date":"2023-05-16","arxiv_id":"2305.09458","n_code_links":1,"syntology":null},{"paper":"/paper/the-muse-2023-multimodal-sentiment-analysis","title":"The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisation","date":"2023-05-05","arxiv_id":"2305.03369","n_code_links":1,"syntology":null},{"paper":"/paper/sportsmot-a-large-multi-object-tracking","title":"SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes","date":"2023-04-11","arxiv_id":"2304.05170","n_code_links":1,"syntology":null},{"paper":"/paper/vars-video-assistant-referee-system-for","title":"VARS: Video Assistant Referee System for Automated Soccer Decision Making from Multiple Views","date":"2023-04-10","arxiv_id":"2304.04617","n_code_links":1,"syntology":null},{"paper":null,"title":"Towards Active Learning for Action Spotting in Association Football Videos","date":"2023-04-09","arxiv_id":"2304.04220","n_code_links":0,"syntology":null},{"paper":"/paper/a-hybrid-multi-scale-attention-convolution","title":"A Hybrid Multi-Scale Attention Convolution and Aging Transformer Network for Alzheimer's Disease Diagnosis","date":"2023-04-07","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"title":"Exemplar-based Video Colorization with Long-term Spatiotemporal Dependency","date":"2023-03-27","arxiv_id":"2303.15081","n_code_links":0,"syntology":null},{"paper":null,"title":"Embedding Contextual Information through Reward Shaping in Multi-Agent Learning: A Case Study from Google Football","date":"2023-03-25","arxiv_id":"2303.15471","n_code_links":0,"syntology":null},{"paper":null,"title":"Deep Generative Multi-Agent Imitation Model as a Computational Benchmark for Evaluating Human Performance in Complex Interactive Tasks: A Case Study in Football","date":"2023-03-23","arxiv_id":"2303.13323","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/image-generation","name":"Image Generation","papers":37},{"task":"/task/conditional-image-generation","name":"Conditional Image Generation","papers":15},{"task":"/task/object-detection","name":"Object Detection","papers":14},{"task":null,"name":"Generative Adversarial Network","papers":12},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":12},{"task":"/task/instance-segmentation","name":"Instance Segmentation","papers":10},{"task":"/task/object-detection-1","name":"object-detection","papers":10},{"task":"/task/segmentation","name":"Segmentation","papers":9},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":9},{"task":"/task/super-resolution","name":"Super-Resolution","papers":8},{"task":"/task/multi-agent-reinforcement-learning","name":"Multi-agent Reinforcement Learning","papers":7},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":7},{"task":"/task/action-recognition-in-videos","name":"Action Recognition","papers":6},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":6},{"task":"/task/denoising","name":"Denoising","papers":6},{"task":"/task/image-classification","name":"Image Classification","papers":6},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":6},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":6},{"task":"/task/image-classification","name":"image-classification","papers":6},{"task":"/task/decision-making","name":"Decision Making","papers":5}],"tasks_shown":20,"n_tasks":197,"usage_by_year":[{"year":"2017","papers":1},{"year":"2018","papers":5},{"year":"2019","papers":20},{"year":"2020","papers":45},{"year":"2021","papers":26},{"year":"2022","papers":36},{"year":"2023","papers":41},{"year":"2024","papers":6},{"year":"2025","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/non-local-operation"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}