{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/side-tuning-network-adaptation-via-additive-1","title":"Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks","arxiv_id":"1912.13503","date":"2019-12-31","proceeding":"ECCV 2020 8","authors":["Jeffrey O. Zhang","Alexander Sax","Amir Zamir","Leonidas Guibas","Jitendra Malik"],"abstract":"When training a neural network for a desired task, one may prefer to adapt a pre-trained network rather than starting from randomly initialized weights. Adaptation can be useful in cases when training data is scarce, when a single learner needs to perform multiple tasks, or when one wishes to encode priors in the network. The most commonly employed approaches for network adaptation are fine-tuning and using the pre-trained network as a fixed feature extractor, among others. In this paper, we propose a straightforward alternative: side-tuning. Side-tuning adapts a pre-trained network by training a lightweight \"side\" network that is fused with the (unchanged) pre-trained network via summation. This simple method works as well as or better than existing solutions and it resolves some of the basic issues with fine-tuning, fixed features, and other common approaches. In particular, side-tuning is less prone to overfitting, is asymptotically consistent, and does not suffer from catastrophic forgetting in incremental learning. We demonstrate the performance of side-tuning under a diverse set of scenarios, including incremental learning (iCIFAR, iTaskonomy), reinforcement learning, imitation learning (visual navigation in Habitat), NLP question-answering (SQuAD v2), and single-task transfer learning (Taskonomy), with consistently promising results.","url_abs":"https://arxiv.org/abs/1912.13503v4","url_pdf":"https://arxiv.org/pdf/1912.13503v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"side-tuning-network-adaptation-via-additive-1","repo_url":"https://github.com/jozhang97/side-tuning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"side-tuning-network-adaptation-via-additive-1","repo_url":"https://github.com/thezingaro/multimodal-side-tuning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[{"task_slug":"imitation-learning","task_name":"Imitation Learning"},{"task_slug":"incremental-learning","task_name":"Incremental Learning"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"visual-navigation","task_name":"Visual Navigation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1912.13503","atlas_url":"https://app.syntology.ai/?focus=1912.13503","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}