{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/xcit-cross-covariance-image-transformers","title":"XCiT: Cross-Covariance Image Transformers","arxiv_id":"2106.09681","date":"2021-06-17","proceeding":"NeurIPS 2021 12","authors":["Alaaeldin El-Nouby","Hugo Touvron","Mathilde Caron","Piotr Bojanowski","Matthijs Douze","Armand Joulin","Ivan Laptev","Natalia Neverova","Gabriel Synnaeve","Jakob Verbeek","Hervé Jegou"],"abstract":"Following their success in natural language processing, transformers have recently shown much promise for computer vision. The self-attention operation underlying transformers yields global interactions between all tokens ,i.e. words or image patches, and enables flexible modelling of image data beyond the local interactions of convolutions. This flexibility, however, comes with a quadratic complexity in time and memory, hindering application to long sequences and high-resolution images. We propose a \"transposed\" version of self-attention that operates across feature channels rather than tokens, where the interactions are based on the cross-covariance matrix between keys and queries. The resulting cross-covariance attention (XCA) has linear complexity in the number of tokens, and allows efficient processing of high-resolution images. Our cross-covariance image transformer (XCiT) is built upon XCA. It combines the accuracy of conventional transformers with the scalability of convolutional architectures. We validate the effectiveness and generality of XCiT by reporting excellent results on multiple vision benchmarks, including image classification and self-supervised feature learning on ImageNet-1k, object detection and instance segmentation on COCO, and semantic segmentation on ADE20k.","url_abs":"https://arxiv.org/abs/2106.09681v2","url_pdf":"https://arxiv.org/pdf/2106.09681v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"xcit-cross-covariance-image-transformers","repo_url":"https://github.com/facebookresearch/xcit","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"xcit-cross-covariance-image-transformers","repo_url":"https://github.com/rwightman/pytorch-image-models","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"xcit-cross-covariance-image-transformers","repo_url":"https://github.com/BrilliantYuKaimin/XCiT-PaddlePaddle","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"paddle","reach":{"status":"ok"}},{"paper_slug":"xcit-cross-covariance-image-transformers","repo_url":"https://github.com/adrienangeli/dino","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"xcit-cross-covariance-image-transformers","repo_url":"https://github.com/facebookresearch/dino","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"xcit-cross-covariance-image-transformers","repo_url":"https://github.com/facebookresearch/vissl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"xcit-cross-covariance-image-transformers","repo_url":"https://github.com/BR-IDL/PaddleViT/tree/develop/image_classification/XCiT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":null},{"paper_slug":"xcit-cross-covariance-image-transformers","repo_url":"https://github.com/MS-Mind/MS-Code-02/tree/main/configs/xcit","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"xcit-cross-covariance-image-transformers","repo_url":"https://github.com/Mind23-2/MindCode-123","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok"}},{"paper_slug":"xcit-cross-covariance-image-transformers","repo_url":"https://github.com/PaddlePaddle/PASSL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"xcit-cross-covariance-image-transformers","repo_url":"https://github.com/alibaba/EasyCV","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"xcit-cross-covariance-image-transformers","repo_url":"https://gitlab.com/birder/birder","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"self-supervised-image-classification","task_name":"Self-Supervised Image Classification"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"cross-covariance-attention","method_name":"Cross-Covariance Attention"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"depthwise-convolution","method_name":"Depthwise Convolution"},{"method_slug":"feedforward-network","method_name":"Feedforward Network"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"local-patch-interaction","method_name":"Local Patch Interaction"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"xcit","method_name":"XCiT"},{"method_slug":"xcit-layer","method_name":"XCiT Layer"}],"datasets_introduced":[],"methods_introduced":[{"slug":"local-patch-interaction","name":"Local Patch Interaction","full_name":"Local Patch Interaction"},{"slug":"xcit","name":"XCiT","full_name":"XCiT"},{"slug":"xcit-layer","name":"XCiT Layer","full_name":"XCiT Layer"}],"results":[{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"XCiT-L24","rank_in_archive_order":182,"of":1060,"metrics":{"GFLOPs":"417.9","Number of params":"189M","Top 1 Accuracy":"86%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"XCiT-M24","rank_in_archive_order":193,"of":1060,"metrics":{"GFLOPs":"188","Number of params":"84M","Top 1 Accuracy":"85.8%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"XCiT-S24","rank_in_archive_order":213,"of":1060,"metrics":{"GFLOPs":"106","Number of params":"48M","Top 1 Accuracy":"85.6%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"XCiT-S12","rank_in_archive_order":256,"of":1060,"metrics":{"GFLOPs":"55.6","Number of params":"26M","Top 1 Accuracy":"85.1%"},"uses_additional_data":false},{"leaderboard":"/sota/instance-segmentation-on-coco-minival","task":"Instance Segmentation","dataset":"COCO minival","model":"XCiT-M24/8","rank_in_archive_order":60,"of":93,"metrics":{"mask AP":"43.7"},"uses_additional_data":false},{"leaderboard":"/sota/instance-segmentation-on-coco-minival","task":"Instance Segmentation","dataset":"COCO minival","model":"XCiT-S24/8","rank_in_archive_order":63,"of":93,"metrics":{"mask AP":"43.0"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-coco-minival","task":"Object Detection","dataset":"COCO minival","model":"XCiT-M24/8","rank_in_archive_order":90,"of":220,"metrics":{"box AP":"48.5"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-coco-minival","task":"Object Detection","dataset":"COCO minival","model":"XCiT-S24/8","rank_in_archive_order":92,"of":220,"metrics":{"box AP":"48.1"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"XCiT-M24/8 (UperNet)","rank_in_archive_order":146,"of":235,"metrics":{"Validation mIoU":"48.4"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"XCiT-S24/8 (UperNet)","rank_in_archive_order":152,"of":235,"metrics":{"Validation mIoU":"48.1"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"XCiT-S24/8 (Semantic-FPN)","rank_in_archive_order":167,"of":235,"metrics":{"Validation mIoU":"47.1"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"XCiT-M24/8 (Semantic-FPN)","rank_in_archive_order":169,"of":235,"metrics":{"Validation mIoU":"46.9"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"XCiT-S12/8 (UperNet)","rank_in_archive_order":174,"of":235,"metrics":{"Validation mIoU":"46.6"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"XCiT-S12/8 (Semantic-FPN)","rank_in_archive_order":204,"of":235,"metrics":{"Validation mIoU":"44.2"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.09681","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.09681"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rwightman/pytorch-image-models","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/dino","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/vissl","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PaddlePaddle/PASSL","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/BR-IDL/PaddleViT/tree/develop/image_classification/XCiT","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/xcit","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alibaba/EasyCV","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://gitlab.com/birder/birder","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MS-Mind/MS-Code-02/tree/main/configs/xcit","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/adrienangeli/dino","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/BrilliantYuKaimin/XCiT-PaddlePaddle","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Mind23-2/MindCode-123","reach":{"status":"ok"}}],"summary":{"ran":2,"ran_honours":1,"unverified":11},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1},"listed":{"samples":11,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"c19231378e41dc5a","entry":"bool_flag","repo":"adrienangeli/dino","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/adrienangeli/dino/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c19231378e41dc5a"}},{"code_sha256_prefix":"2b5489053b222e00","entry":"clip_gradients","repo":"adrienangeli/dino","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/adrienangeli/dino/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2b5489053b222e00"}},{"code_sha256_prefix":"361a6b24f11fc50a","entry":"cosine_scheduler","repo":"adrienangeli/dino","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/adrienangeli/dino/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"361a6b24f11fc50a"}},{"code_sha256_prefix":"18119fb5f5299851","entry":"build_dataset","repo":"facebookresearch/xcit","repo_kind":"official","path":"datasets.py","file_url":"https://github.com/facebookresearch/xcit/blob/HEAD/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"18119fb5f5299851"}},{"code_sha256_prefix":"e7791ea55637b67a","entry":"build_transform","repo":"facebookresearch/xcit","repo_kind":"official","path":"datasets.py","file_url":"https://github.com/facebookresearch/xcit/blob/HEAD/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e7791ea55637b67a"}},{"code_sha256_prefix":"49cf1f4ce52e9613","entry":"config_imname","repo":"adrienangeli/dino","repo_kind":"listed","path":"eval_image_retrieval.py","file_url":"https://github.com/adrienangeli/dino/blob/HEAD/eval_image_retrieval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"49cf1f4ce52e9613"}},{"code_sha256_prefix":"91c5ae2396603107","entry":"config_qimname","repo":"adrienangeli/dino","repo_kind":"listed","path":"eval_image_retrieval.py","file_url":"https://github.com/adrienangeli/dino/blob/HEAD/eval_image_retrieval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"91c5ae2396603107"}},{"code_sha256_prefix":"429abe9dc8793925","entry":"conv3x3","repo":"facebookresearch/xcit","repo_kind":"official","path":"xcit.py","file_url":"https://github.com/facebookresearch/xcit/blob/HEAD/xcit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"429abe9dc8793925"}},{"code_sha256_prefix":"dfebfd56865b6379","entry":"is_image_file","repo":"adrienangeli/dino","repo_kind":"listed","path":"eval_copy_detection.py","file_url":"https://github.com/adrienangeli/dino/blob/HEAD/eval_copy_detection.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dfebfd56865b6379"}},{"code_sha256_prefix":"6ecb6791e6a9ebfb","entry":"knn_classifier","repo":"adrienangeli/dino","repo_kind":"listed","path":"eval_knn.py","file_url":"https://github.com/adrienangeli/dino/blob/HEAD/eval_knn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6ecb6791e6a9ebfb"}},{"code_sha256_prefix":"3093e0466654d790","entry":"label_propagation","repo":"adrienangeli/dino","repo_kind":"listed","path":"eval_video_segmentation.py","file_url":"https://github.com/adrienangeli/dino/blob/HEAD/eval_video_segmentation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3093e0466654d790"}},{"code_sha256_prefix":"e015f2c93c762aff","entry":"norm_mask","repo":"adrienangeli/dino","repo_kind":"listed","path":"eval_video_segmentation.py","file_url":"https://github.com/adrienangeli/dino/blob/HEAD/eval_video_segmentation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e015f2c93c762aff"}},{"code_sha256_prefix":"efc78e753491eeab","entry":"restrict_neighborhood","repo":"adrienangeli/dino","repo_kind":"listed","path":"eval_video_segmentation.py","file_url":"https://github.com/adrienangeli/dino/blob/HEAD/eval_video_segmentation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"efc78e753491eeab"}},{"code_sha256_prefix":"5f21619ff716c1ce","entry":"score_ap_from_ranks_1","repo":"adrienangeli/dino","repo_kind":"listed","path":"eval_copy_detection.py","file_url":"https://github.com/adrienangeli/dino/blob/HEAD/eval_copy_detection.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5f21619ff716c1ce"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}