{"url":"/method/cross-covariance-attention","slug":"cross-covariance-attention","name":"Cross-Covariance Attention","full_name":"Cross-Covariance Attention","full_name_withheld":false,"description_markdown":"**Cross-Covariance Attention**, or **XCA**, is an [attention mechanism](https://paperswithcode.com/methods/category/attention-mechanisms-1) which operates along the feature dimension instead of the token dimension as in [conventional transformers](https://paperswithcode.com/methods/category/transformers).\r\n\r\nUsing the definitions of queries, keys and values from conventional attention, the cross-covariance attention function is defined as:\r\n\r\n$$\r\n\\text { XC-Attention }(Q, K, V)=V \\mathcal{A}_{\\mathrm{XC}}(K, Q), \\quad \\mathcal{A}\\_{\\mathrm{XC}}(K, Q)=\\operatorname{Softmax}\\left(\\hat{K}^{\\top} \\hat{Q} / \\tau\\right)\r\n$$\r\n\r\nwhere each output token embedding is a convex combination of the $d\\_{v}$ features of its corresponding token embedding in $V$. The attention weights $\\mathcal{A}$ are computed based on the cross-covariance matrix.","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"https://arxiv.org/abs/2106.09681v2","title":"XCiT: Cross-Covariance Image Transformers","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Attention Mechanisms","url":"/methods/category/attention-mechanisms","pwc_aliases":["attention-mechanisms-1"]}],"n_papers_tagged":7,"archive_num_papers":null,"papers_newest_first":[{"paper":"/paper/advancing-deformable-medical-image","title":"Advancing Deformable Medical Image Registration with Multi-axis Cross-covariance Attention","date":"2024-12-24","arxiv_id":"2412.18545","n_code_links":1,"syntology":null},{"paper":null,"title":"An I2I Inpainting Approach for Efficient Channel Knowledge Map Construction","date":"2024-06-14","arxiv_id":"2406.09822","n_code_links":0,"syntology":null},{"paper":"/paper/eulermormer-robust-eulerian-motion","title":"EulerMormer: Robust Eulerian Motion Magnification via Dynamic Filtering within Transformer","date":"2023-12-07","arxiv_id":"2312.04152","n_code_links":2,"syntology":null},{"paper":null,"title":"CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Models","date":"2022-10-14","arxiv_id":"2210.09223","n_code_links":0,"syntology":null},{"paper":"/paper/sima-simple-softmax-free-attention-for-vision","title":"SimA: Simple Softmax-free Attention for Vision Transformers","date":"2022-06-17","arxiv_id":"2206.08898","n_code_links":1,"syntology":{"ran":0,"of":3,"unverified":3,"pointer_only":0}},{"paper":"/paper/pe-former-pose-estimation-transformer","title":"PE-former: Pose Estimation Transformer","date":"2021-12-09","arxiv_id":"2112.04981","n_code_links":1,"syntology":null},{"paper":"/paper/xcit-cross-covariance-image-transformers","title":"XCiT: Cross-Covariance Image Transformers","date":"2021-06-17","arxiv_id":"2106.09681","n_code_links":12,"syntology":{"ran":3,"of":14,"unverified":11,"pointer_only":3}}],"papers_shown":7,"tasks":[{"task":"/task/image-classification","name":"Image Classification","papers":3},{"task":"/task/image-classification","name":"image-classification","papers":2},{"task":"/task/decoder","name":"Decoder","papers":1},{"task":"/task/deformable-medical-image-registration","name":"Deformable Medical Image Registration","papers":1},{"task":"/task/denoising","name":"Denoising","papers":1},{"task":"/task/image-registration","name":"Image Registration","papers":1},{"task":"/task/instance-segmentation","name":"Instance Segmentation","papers":1},{"task":"/task/medical-image-analysis","name":"Medical Image Analysis","papers":1},{"task":"/task/medical-image-registration","name":"Medical Image Registration","papers":1},{"task":"/task/motion-magnification","name":"Motion Magnification","papers":1},{"task":"/task/object-detection","name":"Object Detection","papers":1},{"task":"/task/pose-estimation","name":"Pose Estimation","papers":1},{"task":"/task/quantization","name":"Quantization","papers":1},{"task":"/task/self-supervised-image-classification","name":"Self-Supervised Image Classification","papers":1},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":1},{"task":"/task/object-detection-1","name":"object-detection","papers":1}],"tasks_shown":16,"n_tasks":16,"usage_by_year":[{"year":"2021","papers":2},{"year":"2022","papers":2},{"year":"2023","papers":1},{"year":"2024","papers":2}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/cross-covariance-attention"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}