{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/actnn-reducing-training-memory-footprint-via","title":"ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training","arxiv_id":"2104.14129","date":"2021-04-29","proceeding":null,"authors":["Jianfei Chen","Lianmin Zheng","Zhewei Yao","Dequan Wang","Ion Stoica","Michael W. Mahoney","Joseph E. Gonzalez"],"abstract":"The increasing size of neural network models has been critical for improvements in their accuracy, but device memory is not growing at the same rate. This creates fundamental challenges for training neural networks within limited memory environments. In this work, we propose ActNN, a memory-efficient training framework that stores randomly quantized activations for back propagation. We prove the convergence of ActNN for general network architectures, and we characterize the impact of quantization on the convergence via an exact expression for the gradient variance. Using our theory, we propose novel mixed-precision quantization strategies that exploit the activation's heterogeneity across feature dimensions, samples, and layers. These techniques can be readily applied to existing dynamic graph frameworks, such as PyTorch, simply by substituting the layers. We evaluate ActNN on mainstream computer vision models for classification, detection, and segmentation tasks. On all these tasks, ActNN compresses the activation to 2 bits on average, with negligible accuracy loss. ActNN reduces the memory footprint of the activation by 12x, and it enables training with a 6.6x to 14x larger batch size.","url_abs":"https://arxiv.org/abs/2104.14129v2","url_pdf":"https://arxiv.org/pdf/2104.14129v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"actnn-reducing-training-memory-footprint-via","repo_url":"https://github.com/ucbrise/actnn","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"actnn-reducing-training-memory-footprint-via","repo_url":"https://github.com/guanchuwang/division","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"actnn-reducing-training-memory-footprint-via","repo_url":"https://github.com/warai-0toko/exact","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"actnn-reducing-training-memory-footprint-via","repo_url":"https://github.com/zirui-ray-liu/exact","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2104.14129","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.14129"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ucbrise/actnn","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/warai-0toko/exact","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/guanchuwang/division","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zirui-ray-liu/exact","reach":{"status":"ok"}}],"summary":{"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"91fa51701824ba48","entry":"get_memory_usage","repo":"ucbrise/actnn","repo_kind":"official","path":"actnn/actnn/utils.py","file_url":"https://github.com/ucbrise/actnn/blob/HEAD/actnn/actnn/utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91fa51701824ba48"}},{"code_sha256_prefix":"0088baa4bd06a22e","entry":"compute_tensor_bytes","repo":"ucbrise/actnn","repo_kind":"official","path":"actnn/actnn/utils.py","file_url":"https://github.com/ucbrise/actnn/blob/HEAD/actnn/actnn/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0088baa4bd06a22e"}},{"code_sha256_prefix":"4c4e32743ef5c91f","entry":"swap_to_cpu","repo":"ucbrise/actnn","repo_kind":"official","path":"actnn/actnn/utils.py","file_url":"https://github.com/ucbrise/actnn/blob/HEAD/actnn/actnn/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4c4e32743ef5c91f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}