{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/textrm-d-3-textrm-former-debiased-dual","title":"D3Former: Debiased Dual Distilled Transformer for Incremental Learning","arxiv_id":"2208.00777","date":"2022-07-25","proceeding":null,"authors":["Abdelrahman Mohamed","Rushali Grandhe","K J Joseph","Salman Khan","Fahad Khan"],"abstract":"In class incremental learning (CIL) setting, groups of classes are introduced to a model in each learning phase. The goal is to learn a unified model performant on all the classes observed so far. Given the recent popularity of Vision Transformers (ViTs) in conventional classification settings, an interesting question is to study their continual learning behaviour. In this work, we develop a Debiased Dual Distilled Transformer for CIL dubbed $\\textrm{D}^3\\textrm{Former}$. The proposed model leverages a hybrid nested ViT design to ensure data efficiency and scalability to small as well as large datasets. In contrast to a recent ViT based CIL approach, our $\\textrm{D}^3\\textrm{Former}$ does not dynamically expand its architecture when new tasks are learned and remains suitable for a large number of incremental tasks. The improved CIL behaviour of $\\textrm{D}^3\\textrm{Former}$ owes to two fundamental changes to the ViT design. First, we treat the incremental learning as a long-tail classification problem where the majority samples from new classes vastly outnumber the limited exemplars available for old classes. To avoid the bias against the minority old classes, we propose to dynamically adjust logits to emphasize on retaining the representations relevant to old tasks. Second, we propose to preserve the configuration of spatial attention maps as the learning progresses across tasks. This helps in reducing catastrophic forgetting by constraining the model to retain the attention on the most discriminative regions. $\\textrm{D}^3\\textrm{Former}$ obtains favorable results on incremental versions of CIFAR-100, MNIST, SVHN, and ImageNet datasets. Code is available at https://tinyurl.com/d3former","url_abs":"https://arxiv.org/abs/2208.00777v3","url_pdf":"https://arxiv.org/pdf/2208.00777v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"textrm-d-3-textrm-former-debiased-dual","repo_url":"https://github.com/abdohelmy/D-3Former","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":"incremental-learning","task_name":"Incremental Learning"},{"task_slug":"class-incremental-learning-1","task_name":"class-incremental learning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/incremental-learning-on-cifar-100-50-classes-2","task":"Incremental Learning","dataset":"CIFAR-100 - 50 classes + 10 steps of 5 classes","model":"D3Former","rank_in_archive_order":4,"of":13,"metrics":{"Average Incremental Accuracy":"70.94"},"uses_additional_data":false},{"leaderboard":"/sota/incremental-learning-on-cifar-100-50-classes-1","task":"Incremental Learning","dataset":"CIFAR-100 - 50 classes + 25 steps of 2 classes","model":"D3Former","rank_in_archive_order":1,"of":5,"metrics":{"Average Incremental Accuracy":"68.68"},"uses_additional_data":false},{"leaderboard":"/sota/incremental-learning-on-cifar-100-50-classes-3","task":"Incremental Learning","dataset":"CIFAR-100 - 50 classes + 5 steps of 10 classes","model":"D3Former","rank_in_archive_order":4,"of":15,"metrics":{"Average Incremental Accuracy":"72.23"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2208.00777","atlas_url":"https://app.syntology.ai/?focus=2208.00777","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}