{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/elliptical-attention","title":"Elliptical Attention","arxiv_id":"2406.13770","date":"2024-06-19","proceeding":null,"authors":["Stefan K. Nielsen","Laziz U. Abdullaev","Rachel S. Y. Teo","Tan M. Nguyen"],"abstract":"Pairwise dot-product self-attention is key to the success of transformers that achieve state-of-the-art performance across a variety of applications in language and vision. This dot-product self-attention computes attention weights among the input tokens using Euclidean distance, which makes the model prone to representation collapse and vulnerable to contaminated samples. In this paper, we propose using a Mahalanobis distance metric for computing the attention weights to stretch the underlying feature space in directions of high contextual relevance. In particular, we define a hyper-ellipsoidal neighborhood around each query to increase the attention weights of the tokens lying in the contextually important directions. We term this novel class of attention Elliptical Attention. Our Elliptical Attention provides two benefits: 1) reducing representation collapse and 2) enhancing the model's robustness as Elliptical Attention pays more attention to contextually relevant information rather than focusing on some small subset of informative features. We empirically demonstrate the advantages of Elliptical Attention over the baseline dot-product attention and state-of-the-art attention methods on various practical tasks, including object classification, image segmentation, and language modeling across different data modalities.","url_abs":"https://arxiv.org/abs/2406.13770v2","url_pdf":"https://arxiv.org/pdf/2406.13770v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"elliptical-attention","repo_url":"https://github.com/stefvk/elliptical-attention","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.13770","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.13770"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/stefvk/Elliptical-Attention","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/stefvk/elliptical-attention","reach":{"status":"ok"}}],"summary":{"ran":2,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"1c4396c7bf90ccbb","entry":"create_exp_dir","repo":"stefvk/Elliptical-Attention","repo_kind":"official","path":"ImageNet/logger.py","file_url":"https://github.com/stefvk/Elliptical-Attention/blob/HEAD/ImageNet/logger.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1c4396c7bf90ccbb"}},{"code_sha256_prefix":"d142e8a68b535885","entry":"get_logger","repo":"stefvk/Elliptical-Attention","repo_kind":"official","path":"ImageNet/logger.py","file_url":"https://github.com/stefvk/Elliptical-Attention/blob/HEAD/ImageNet/logger.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d142e8a68b535885"}},{"code_sha256_prefix":"71210cfed12ff00a","entry":"MahalaAttention","repo":"stefvk/elliptical-attention","repo_kind":"official","path":"ADE20K/model/deit_elliptical.py","file_url":"https://github.com/stefvk/elliptical-attention/blob/HEAD/ADE20K/model/deit_elliptical.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"71210cfed12ff00a"}},{"code_sha256_prefix":"9153241c4331eb1a","entry":"build_dataset","repo":"stefvk/Elliptical-Attention","repo_kind":"official","path":"ImageNet/datasets.py","file_url":"https://github.com/stefvk/Elliptical-Attention/blob/HEAD/ImageNet/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9153241c4331eb1a"}},{"code_sha256_prefix":"3bc74137ab36aa79","entry":"build_transform","repo":"stefvk/Elliptical-Attention","repo_kind":"official","path":"ImageNet/datasets.py","file_url":"https://github.com/stefvk/Elliptical-Attention/blob/HEAD/ImageNet/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3bc74137ab36aa79"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}