{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tiny-transfer-learning-towards-memory","title":"TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning","arxiv_id":"2007.11622","date":"2020-07-22","proceeding":"NeurIPS 2020 12","authors":["Han Cai","Chuang Gan","Ligeng Zhu","Song Han"],"abstract":"On-device learning enables edge devices to continually adapt the AI models to new data, which requires a small memory footprint to fit the tight memory constraint of edge devices. Existing work solves this problem by reducing the number of trainable parameters. However, this doesn't directly translate to memory saving since the major bottleneck is the activations, not parameters. In this work, we present Tiny-Transfer-Learning (TinyTL) for memory-efficient on-device learning. TinyTL freezes the weights while only learns the bias modules, thus no need to store the intermediate activations. To maintain the adaptation capacity, we introduce a new memory-efficient bias module, the lite residual module, to refine the feature extractor by learning small residual feature maps adding only 3.8% memory overhead. Extensive experiments show that TinyTL significantly saves the memory (up to 6.5x) with little accuracy loss compared to fine-tuning the full network. Compared to fine-tuning the last layer, TinyTL provides significant accuracy improvements (up to 34.1%) with little memory overhead. Furthermore, combined with feature extractor adaptation, TinyTL provides 7.3-12.9x memory saving without sacrificing accuracy compared to fine-tuning the full Inception-V3.","url_abs":"https://arxiv.org/abs/2007.11622v5","url_pdf":"https://arxiv.org/pdf/2007.11622v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tiny-transfer-learning-towards-memory","repo_url":"https://github.com/mit-han-lab/tinyML","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2007.11622","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2007.11622"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mit-han-lab/tinyML","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":10},"by_repo_kind":{"official":{"samples":10,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7b60f79eb05538dd","entry":"accuracy","repo":"mit-han-lab/tinyML","repo_kind":"official","path":"netaug/utils/metric.py","file_url":"https://github.com/mit-han-lab/tinyML/blob/HEAD/netaug/utils/metric.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b60f79eb05538dd"}},{"code_sha256_prefix":"2f3a2114fb7ba02f","entry":"build_act","repo":"mit-han-lab/tinyML","repo_kind":"official","path":"netaug/models/base/layers.py","file_url":"https://github.com/mit-han-lab/tinyML/blob/HEAD/netaug/models/base/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f3a2114fb7ba02f"}},{"code_sha256_prefix":"c30d44c8efdb7774","entry":"build_norm","repo":"mit-han-lab/tinyML","repo_kind":"official","path":"netaug/models/base/layers.py","file_url":"https://github.com/mit-han-lab/tinyML/blob/HEAD/netaug/models/base/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c30d44c8efdb7774"}},{"code_sha256_prefix":"652ec8594912a100","entry":"get_module_device","repo":"mit-han-lab/tinyML","repo_kind":"official","path":"netaug/utils/profile.py","file_url":"https://github.com/mit-han-lab/tinyML/blob/HEAD/netaug/utils/profile.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"652ec8594912a100"}},{"code_sha256_prefix":"b0d3762fc8297376","entry":"is_parallel","repo":"mit-han-lab/tinyML","repo_kind":"official","path":"netaug/utils/profile.py","file_url":"https://github.com/mit-han-lab/tinyML/blob/HEAD/netaug/utils/profile.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0d3762fc8297376"}},{"code_sha256_prefix":"73cb2dc75f25569d","entry":"label_smooth","repo":"mit-han-lab/tinyML","repo_kind":"official","path":"netaug/utils/criterion.py","file_url":"https://github.com/mit-han-lab/tinyML/blob/HEAD/netaug/utils/criterion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73cb2dc75f25569d"}},{"code_sha256_prefix":"7b1fae01307631bc","entry":"list_sum","repo":"mit-han-lab/tinyML","repo_kind":"official","path":"netaug/utils/misc.py","file_url":"https://github.com/mit-han-lab/tinyML/blob/HEAD/netaug/utils/misc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b1fae01307631bc"}},{"code_sha256_prefix":"8466bd9906056233","entry":"load_state_dict_from_file","repo":"mit-han-lab/tinyML","repo_kind":"official","path":"netaug/utils/misc.py","file_url":"https://github.com/mit-han-lab/tinyML/blob/HEAD/netaug/utils/misc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8466bd9906056233"}},{"code_sha256_prefix":"e54f2057f08d98b0","entry":"make_divisible","repo":"mit-han-lab/tinyML","repo_kind":"official","path":"netaug/utils/misc.py","file_url":"https://github.com/mit-han-lab/tinyML/blob/HEAD/netaug/utils/misc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e54f2057f08d98b0"}},{"code_sha256_prefix":"74db851166d0f374","entry":"trainable_param_num","repo":"mit-han-lab/tinyML","repo_kind":"official","path":"netaug/utils/profile.py","file_url":"https://github.com/mit-han-lab/tinyML/blob/HEAD/netaug/utils/profile.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74db851166d0f374"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}