{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-exemplar-free-continual-learning-in","title":"Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularization","arxiv_id":"2203.13167","date":"2022-03-24","proceeding":null,"authors":["Francesco Pelosin","Saurav Jha","Andrea Torsello","Bogdan Raducanu","Joost Van de Weijer"],"abstract":"In this paper, we investigate the continual learning of Vision Transformers (ViT) for the challenging exemplar-free scenario, with special focus on how to efficiently distill the knowledge of its crucial self-attention mechanism (SAM). Our work takes an initial step towards a surgical investigation of SAM for designing coherent continual learning methods in ViTs. We first carry out an evaluation of established continual learning regularization techniques. We then examine the effect of regularization when applied to two key enablers of SAM: (a) the contextualized embedding layers, for their ability to capture well-scaled representations with respect to the values, and (b) the prescaled attention maps, for carrying value-independent global contextual information. We depict the perks of each distilling strategy on two image recognition benchmarks (CIFAR100 and ImageNet-32) -- while (a) leads to a better overall accuracy, (b) helps enhance the rigidity by maintaining competitive performances. Furthermore, we identify the limitation imposed by the symmetric nature of regularization losses. To alleviate this, we propose an asymmetric variant and apply it to the pooled output distillation (POD) loss adapted for ViTs. Our experiments confirm that introducing asymmetry to POD boosts its plasticity while retaining stability across (a) and (b). Moreover, we acknowledge low forgetting measures for all the compared methods, indicating that ViTs might be naturally inclined continual learner","url_abs":"https://arxiv.org/abs/2203.13167v4","url_pdf":"https://arxiv.org/pdf/2203.13167v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-exemplar-free-continual-learning-in","repo_url":"https://github.com/srvCodes/continual_learning_with_vit","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":"exemplar-free","task_name":"Exemplar-Free"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2203.13167","atlas_url":"https://app.syntology.ai/?focus=2203.13167","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.13167"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/srvCodes/continual_learning_with_vit","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"ran_violates":1,"ran_fixture":1,"unverified":4},"by_repo_kind":{"official":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2e179edce79cd470","entry":"last_layer_analysis","repo":"srvCodes/continual_learning_with_vit","repo_kind":"official","path":"src/last_layer_analysis.py","file_url":"https://github.com/srvCodes/continual_learning_with_vit/blob/HEAD/src/last_layer_analysis.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e179edce79cd470"}},{"code_sha256_prefix":"3c90ff78f2a788f9","entry":"make_tuple","repo":"srvCodes/continual_learning_with_vit","repo_kind":"official","path":"src/networks/early_conv_vit_net.py","file_url":"https://github.com/srvCodes/continual_learning_with_vit/blob/HEAD/src/networks/early_conv_vit_net.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c90ff78f2a788f9"}},{"code_sha256_prefix":"191b2735955d18f4","entry":"resize_pos_embed","repo":"srvCodes/continual_learning_with_vit","repo_kind":"official","path":"src/networks/vit_original.py","file_url":"https://github.com/srvCodes/continual_learning_with_vit/blob/HEAD/src/networks/vit_original.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"191b2735955d18f4"}},{"code_sha256_prefix":"6d8a1f8a0697180f","entry":"checkpoint_filter_fn","repo":"srvCodes/continual_learning_with_vit","repo_kind":"official","path":"src/networks/vit_original.py","file_url":"https://github.com/srvCodes/continual_learning_with_vit/blob/HEAD/src/networks/vit_original.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d8a1f8a0697180f"}},{"code_sha256_prefix":"504704958c749975","entry":"early_conv_vit","repo":"srvCodes/continual_learning_with_vit","repo_kind":"official","path":"src/networks/early_conv_vit.py","file_url":"https://github.com/srvCodes/continual_learning_with_vit/blob/HEAD/src/networks/early_conv_vit.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"504704958c749975"}},{"code_sha256_prefix":"3d67dffbc87a8dfd","entry":"timm_vit_tiny_16_augreg_224","repo":"srvCodes/continual_learning_with_vit","repo_kind":"official","path":"src/networks/timm_vit_tiny_16_augreg_224.py","file_url":"https://github.com/srvCodes/continual_learning_with_vit/blob/HEAD/src/networks/timm_vit_tiny_16_augreg_224.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d67dffbc87a8dfd"}},{"code_sha256_prefix":"bb773eb0305e934b","entry":"vit_tiny_16_augreg_224","repo":"srvCodes/continual_learning_with_vit","repo_kind":"official","path":"src/networks/vit_tiny_16_augreg_224.py","file_url":"https://github.com/srvCodes/continual_learning_with_vit/blob/HEAD/src/networks/vit_tiny_16_augreg_224.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb773eb0305e934b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}