{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-effectiveness-of-layernorm-tuning-for","title":"On the Effectiveness of LayerNorm Tuning for Continual Learning in Vision Transformers","arxiv_id":"2308.09610","date":"2023-08-18","proceeding":null,"authors":["Thomas De Min","Massimiliano Mancini","Karteek Alahari","Xavier Alameda-Pineda","Elisa Ricci"],"abstract":"State-of-the-art rehearsal-free continual learning methods exploit the peculiarities of Vision Transformers to learn task-specific prompts, drastically reducing catastrophic forgetting. However, there is a tradeoff between the number of learned parameters and the performance, making such models computationally expensive. In this work, we aim to reduce this cost while maintaining competitive performance. We achieve this by revisiting and extending a simple transfer learning idea: learning task-specific normalization layers. Specifically, we tune the scale and bias parameters of LayerNorm for each continual learning task, selecting them at inference time based on the similarity between task-specific keys and the output of the pre-trained model. To make the classifier robust to incorrect selection of parameters during inference, we introduce a two-stage training procedure, where we first optimize the task-specific parameters and then train the classifier with the same selection procedure of the inference time. Experiments on ImageNet-R and CIFAR-100 show that our method achieves results that are either superior or on par with {the state of the art} while being computationally cheaper.","url_abs":"https://arxiv.org/abs/2308.09610v1","url_pdf":"https://arxiv.org/pdf/2308.09610v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-effectiveness-of-layernorm-tuning-for","repo_url":"https://github.com/tdemin16/continual-layernorm-tuning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2308.09610","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.09610"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tdemin16/continual-layernorm-tuning","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":12,"unverified":1},"by_repo_kind":{"official":{"samples":13,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5c3574e986df9984","entry":"calculate_md5","repo":"tdemin16/continual-layernorm-tuning","repo_kind":"official","path":"couple/continual_datasets/dataset_utils.py","file_url":"https://github.com/tdemin16/continual-layernorm-tuning/blob/HEAD/couple/continual_datasets/dataset_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c3574e986df9984"}},{"code_sha256_prefix":"46b62ccc9e5b9878","entry":"check_integrity","repo":"tdemin16/continual-layernorm-tuning","repo_kind":"official","path":"couple/continual_datasets/dataset_utils.py","file_url":"https://github.com/tdemin16/continual-layernorm-tuning/blob/HEAD/couple/continual_datasets/dataset_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"46b62ccc9e5b9878"}},{"code_sha256_prefix":"b3cb500b2f04323d","entry":"check_md5","repo":"tdemin16/continual-layernorm-tuning","repo_kind":"official","path":"couple/continual_datasets/dataset_utils.py","file_url":"https://github.com/tdemin16/continual-layernorm-tuning/blob/HEAD/couple/continual_datasets/dataset_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b3cb500b2f04323d"}},{"code_sha256_prefix":"2b5dd89802152360","entry":"checkpoint_filter_fn","repo":"tdemin16/continual-layernorm-tuning","repo_kind":"official","path":"couple/vision_transformer.py","file_url":"https://github.com/tdemin16/continual-layernorm-tuning/blob/HEAD/couple/vision_transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2b5dd89802152360"}},{"code_sha256_prefix":"6c600a2f0f2361f1","entry":"get_init_weights_vit","repo":"tdemin16/continual-layernorm-tuning","repo_kind":"official","path":"couple/vision_transformer.py","file_url":"https://github.com/tdemin16/continual-layernorm-tuning/blob/HEAD/couple/vision_transformer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6c600a2f0f2361f1"}},{"code_sha256_prefix":"f0220ff4f9a3c317","entry":"get_optimizer","repo":"tdemin16/continual-layernorm-tuning","repo_kind":"official","path":"couple/utils.py","file_url":"https://github.com/tdemin16/continual-layernorm-tuning/blob/HEAD/couple/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f0220ff4f9a3c317"}},{"code_sha256_prefix":"8cdb71f9c1b524f8","entry":"is_trainable","repo":"tdemin16/continual-layernorm-tuning","repo_kind":"official","path":"couple/utils.py","file_url":"https://github.com/tdemin16/continual-layernorm-tuning/blob/HEAD/couple/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8cdb71f9c1b524f8"}},{"code_sha256_prefix":"3dd88ba90d737da4","entry":"mask_logits","repo":"tdemin16/continual-layernorm-tuning","repo_kind":"official","path":"couple/utils.py","file_url":"https://github.com/tdemin16/continual-layernorm-tuning/blob/HEAD/couple/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3dd88ba90d737da4"}},{"code_sha256_prefix":"f1a58177a84f9ec9","entry":"non_negative_float","repo":"tdemin16/continual-layernorm-tuning","repo_kind":"official","path":"couple/configs/custom_types.py","file_url":"https://github.com/tdemin16/continual-layernorm-tuning/blob/HEAD/couple/configs/custom_types.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f1a58177a84f9ec9"}},{"code_sha256_prefix":"321424d80e3f6eed","entry":"resize_pos_embed","repo":"tdemin16/continual-layernorm-tuning","repo_kind":"official","path":"couple/vision_transformer.py","file_url":"https://github.com/tdemin16/continual-layernorm-tuning/blob/HEAD/couple/vision_transformer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"321424d80e3f6eed"}},{"code_sha256_prefix":"cad22a420d442cb1","entry":"restricted_float","repo":"tdemin16/continual-layernorm-tuning","repo_kind":"official","path":"couple/configs/custom_types.py","file_url":"https://github.com/tdemin16/continual-layernorm-tuning/blob/HEAD/couple/configs/custom_types.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cad22a420d442cb1"}},{"code_sha256_prefix":"265ea374aab10001","entry":"target_transform","repo":"tdemin16/continual-layernorm-tuning","repo_kind":"official","path":"couple/datasets.py","file_url":"https://github.com/tdemin16/continual-layernorm-tuning/blob/HEAD/couple/datasets.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"265ea374aab10001"}},{"code_sha256_prefix":"b507dddcf36a89ad","entry":"get_dataset","repo":"tdemin16/continual-layernorm-tuning","repo_kind":"official","path":"couple/datasets.py","file_url":"https://github.com/tdemin16/continual-layernorm-tuning/blob/HEAD/couple/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b507dddcf36a89ad"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}