{"url":"/method/ca","slug":"ca","name":"Class Attention","full_name":"Class Attention","full_name_withheld":false,"description_markdown":"A **Class Attention** layer, or **CA Layer**, is an [attention mechanism](https://paperswithcode.com/methods/category/attention-mechanisms-1) for [vision transformers](https://paperswithcode.com/methods/category/vision-transformer) used in [CaiT](https://paperswithcode.com/method/cait) that aims to extract information from a set of processed patches. It is identical to a [self-attention layer](https://paperswithcode.com/method/scaled), except that it relies on the attention between (i) the class embedding $x_{\\text {class }}$ (initialized at CLS in the first CA) and (ii) itself plus the set of frozen patch embeddings $x_{\\text {patches }} .$ \r\n\r\nConsidering a network with $h$ heads and $p$ patches, and denoting by $d$ the embedding size, the multi-head class-attention is parameterized with several projection matrices, $W_{q}, W_{k}, W_{v}, W_{o} \\in \\mathbf{R}^{d \\times d}$, and the corresponding biases $b_{q}, b_{k}, b_{v}, b_{o} \\in \\mathbf{R}^{d} .$ With this notation, the computation of the CA residual block proceeds as follows. We first augment the patch embeddings (in matrix form) as $z=\\left[x_{\\text {class }}, x_{\\text {patches }}\\right]$. We then perform the projections:\r\n\r\n$$Q=W\\_{q} x\\_{\\text {class }}+b\\_{q}$$\r\n\r\n$$K=W\\_{k} z+b\\_{k}$$\r\n\r\n$$V=W\\_{v} z+b\\_{v}$$\r\n\r\nThe class-attention weights are given by\r\n\r\n$$\r\nA=\\operatorname{Softmax}\\left(Q . K^{T} / \\sqrt{d / h}\\right)\r\n$$\r\n\r\nwhere $Q . K^{T} \\in \\mathbf{R}^{h \\times 1 \\times p}$. This attention is involved in the weighted sum $A \\times V$ to produce the residual output vector\r\n\r\n$$\r\n\\operatorname{out}\\_{\\mathrm{CA}}=W\\_{o} A V+b\\_{o}\r\n$$\r\n\r\nwhich is in turn added to $x\\_{\\text {class }}$ for subsequent processing.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Going deeper with Image Transformers","paper":"/paper/going-deeper-with-image-transformers","first_author":"Hugo Touvron","n_authors":5,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/going-deeper-with-image-transformers"},"source":{"url":"https://arxiv.org/abs/2103.17239v2","title":"Going deeper with Image Transformers","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Attention Mechanisms","url":"/methods/category/attention-mechanisms","pwc_aliases":["attention-mechanisms-1"]},{"area":"General","area_id":"general","collection":"Attention","url":"/methods/category/attention","pwc_aliases":[]}],"n_papers_tagged":36,"archive_num_papers":36,"papers_newest_first":[{"paper":"/paper/prompt-cam-a-simpler-interpretable","title":"Prompt-CAM: A Simpler Interpretable Transformer for Fine-Grained Analysis","date":"2025-01-16","arxiv_id":"2501.09333","n_code_links":1,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":0}},{"paper":"/paper/prompt-cam-making-vision-transformers","title":"Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis","date":"2025-01-01","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/spectralkd-understanding-and-optimizing","title":"SpectralKD: A Unified Framework for Interpreting and Distilling Vision Transformers via Spectral Analysis","date":"2024-12-26","arxiv_id":"2412.19055","n_code_links":1,"syntology":null},{"paper":"/paper/an-attention-based-representation","title":"An Attention-based Representation Distillation Baseline for Multi-Label Continual Learning","date":"2024-07-19","arxiv_id":"2407.14249","n_code_links":1,"syntology":null},{"paper":"/paper/lf-vit-reducing-spatial-redundancy-in-vision","title":"LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition","date":"2024-01-08","arxiv_id":"2402.00033","n_code_links":1,"syntology":{"ran":10,"of":18,"unverified":8,"pointer_only":18}},{"paper":null,"title":"Self-distilled Masked Attention guided masked image modeling with noise Regularized Teacher (SMART) for medical image analysis","date":"2023-10-02","arxiv_id":"2310.01209","n_code_links":0,"syntology":null},{"paper":"/paper/class-attention-transfer-based-knowledge","title":"Class Attention Transfer Based Knowledge Distillation","date":"2023-04-25","arxiv_id":"2304.12777","n_code_links":1,"syntology":{"ran":4,"of":4,"unverified":0,"pointer_only":4}},{"paper":"/paper/sacanet-scene-aware-class-attention-network","title":"SACANet: scene-aware class attention network for semantic segmentation of remote sensing images","date":"2023-04-22","arxiv_id":"2304.11424","n_code_links":1,"syntology":null},{"paper":null,"title":"Detecting Severity of Diabetic Retinopathy from Fundus Images: A Transformer Network-based Review","date":"2023-01-03","arxiv_id":"2301.00973","n_code_links":0,"syntology":null},{"paper":null,"title":"Bidirectional Representations for Low Resource Spoken Language Understanding","date":"2022-11-24","arxiv_id":"2211.14320","n_code_links":0,"syntology":null},{"paper":"/paper/adaptive-sparse-vit-towards-learnable","title":"Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention","date":"2022-09-28","arxiv_id":"2209.13802","n_code_links":1,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":1}},{"paper":"/paper/class-attention-video-transformer-for","title":"Class-attention Video Transformer for Engagement Intensity Prediction","date":"2022-08-12","arxiv_id":"2208.07216","n_code_links":1,"syntology":null},{"paper":null,"title":"MaiT: Leverage Attention Masks for More Efficient Image Transformers","date":"2022-07-06","arxiv_id":"2207.03006","n_code_links":0,"syntology":null},{"paper":"/paper/augmenting-convolutional-networks-with","title":"Augmenting Convolutional networks with attention-based aggregation","date":"2021-12-27","arxiv_id":"2112.13692","n_code_links":5,"syntology":{"ran":1,"of":2,"unverified":1,"pointer_only":0}},{"paper":null,"title":"SSA: Semantic Structure Aware Inference for Weakly Pixel-Wise Dense Predictions without Cost","date":"2021-11-05","arxiv_id":"2111.03392","n_code_links":0,"syntology":null},{"paper":"/paper/mait-integrating-spatial-locality-into-image","title":"MaiT: integrating spatial locality into image transformers with attention masks","date":"2021-09-29","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"title":"Is cell segregation like oil and water: asymptotic versus transitory regime","date":"2021-09-01","arxiv_id":"2109.00364","n_code_links":0,"syntology":null},{"paper":"/paper/dynamic-relevance-learning-for-few-shot","title":"Dynamic Relevance Learning for Few-Shot Object Detection","date":"2021-08-04","arxiv_id":"2108.02235","n_code_links":1,"syntology":null},{"paper":"/paper/evo-vit-slow-fast-token-evolution-for-dynamic","title":"Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer","date":"2021-08-03","arxiv_id":"2108.01390","n_code_links":1,"syntology":{"ran":5,"of":9,"unverified":4,"pointer_only":0}},{"paper":null,"title":"A comparison of latent semantic analysis and correspondence analysis of document-term matrices","date":"2021-07-25","arxiv_id":"2108.06197","n_code_links":0,"syntology":null},{"paper":null,"title":"Otimizacao de Redes Neurais atraves de Algoritmos Geneticos Celulares","date":"2021-07-18","arxiv_id":"2107.08326","n_code_links":0,"syntology":null},{"paper":null,"title":"Scientia Potentia Est -- On the Role of Knowledge in Computational Argumentation","date":"2021-07-01","arxiv_id":"2107.00281","n_code_links":0,"syntology":null},{"paper":null,"title":"Detecting Cattle and Elk in the Wild from Space","date":"2021-06-29","arxiv_id":"2106.15448","n_code_links":0,"syntology":null},{"paper":"/paper/towards-self-organized-control-using-neural","title":"Towards self-organized control: Using neural cellular automata to robustly control a cart-pole agent","date":"2021-06-29","arxiv_id":"2106.15240","n_code_links":1,"syntology":null},{"paper":null,"title":"Contrastive Attention for Automatic Chest X-ray Report Generation","date":"2021-06-13","arxiv_id":"2106.06965","n_code_links":0,"syntology":null},{"paper":"/paper/a-reversible-system-based-on-hybrid-toggle","title":"A reversible system based on hybrid toggle radius-4 cellular automata and its application as a block cipher","date":"2021-06-09","arxiv_id":"2106.04777","n_code_links":1,"syntology":null},{"paper":"/paper/cherry-picking-gradients-learning-low-rank","title":"Cherry-Picking Gradients: Learning Low-Rank Embeddings of Visual Data via Differentiable Cross-Approximation","date":"2021-05-29","arxiv_id":"2105.14250","n_code_links":1,"syntology":null},{"paper":"/paper/evolutionary-algorithms-for-designing","title":"Evolutionary Algorithms for Designing Reversible Cellular Automata","date":"2021-05-25","arxiv_id":"2105.12039","n_code_links":1,"syntology":null},{"paper":null,"title":"Designing AI-based Conversational Agent for Diabetes Care in a Multilingual Context","date":"2021-05-20","arxiv_id":"2105.09490","n_code_links":0,"syntology":null},{"paper":null,"title":"EBM-Fold: Fully-Differentiable Protein Folding Powered by Energy-based Models","date":"2021-05-11","arxiv_id":"2105.04771","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/image-classification","name":"Image Classification","papers":3},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":3},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":3},{"task":"/task/efficient-vits","name":"Efficient ViTs","papers":2},{"task":"/task/knowledge-distillation","name":"Knowledge Distillation","papers":2},{"task":"/task/natural-language-understanding","name":"Natural Language Understanding","papers":2},{"task":"/task/object-detection","name":"Object Detection","papers":2},{"task":"/task/visual-prompt-tuning","name":"Visual Prompt Tuning","papers":2},{"task":"/task/image-classification","name":"image-classification","papers":2},{"task":"/task/authorship-attribution","name":"Authorship Attribution","papers":1},{"task":"/task/automatic-speech-recognition-2","name":"Automatic Speech Recognition","papers":1},{"task":"/task/automatic-speech-recognition","name":"Automatic Speech Recognition (ASR)","papers":1},{"task":"/task/machine-learning","name":"BIG-bench Machine Learning","papers":1},{"task":"/task/classification-1","name":"Classification","papers":1},{"task":"/task/common-sense-reasoning","name":"Common Sense Reasoning","papers":1},{"task":"/task/computed-tomography-ct","name":"Computed Tomography (CT)","papers":1},{"task":"/task/continual-learning","name":"Continual Learning","papers":1},{"task":"/task/decision-making","name":"Decision Making","papers":1},{"task":"/task/denoising","name":"Denoising","papers":1},{"task":"/task/dimensionality-reduction","name":"Dimensionality Reduction","papers":1}],"tasks_shown":20,"n_tasks":49,"usage_by_year":[{"year":"2021","papers":23},{"year":"2022","papers":4},{"year":"2023","papers":4},{"year":"2024","papers":3},{"year":"2025","papers":2}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/ca"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}