{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mesa-a-memory-saving-training-framework-for","title":"Mesa: A Memory-saving Training Framework for Transformers","arxiv_id":"2111.11124","date":"2021-11-22","proceeding":null,"authors":["Zizheng Pan","Peng Chen","Haoyu He","Jing Liu","Jianfei Cai","Bohan Zhuang"],"abstract":"There has been an explosion of interest in designing high-performance Transformers. While Transformers have delivered significant performance improvements, training such networks is extremely memory intensive owing to storing all intermediate activations that are needed for gradient computation during backpropagation, especially for long sequences. To this end, we present Mesa, a memory-saving training framework for Transformers. Specifically, Mesa uses exact activations during forward pass while storing a low-precision version of activations to reduce memory consumption during training. The low-precision activations are then dequantized during back-propagation to compute gradients. Besides, to address the heterogeneous activation distributions in the multi-head self-attention layers, we propose a head-wise activation quantization strategy, which quantizes activations based on the statistics of each head to minimize the approximation error. To further boost training efficiency, we learn quantization parameters by running estimates. More importantly, by re-investing the saved memory in employing a larger batch size or scaling up model size, we may further improve the performance under constrained computational resources. Extensive experiments on ImageNet, CIFAR-100 and ADE20K demonstrate that Mesa can achieve flexible memory-savings (up to 50%) during training while achieving comparable or even better performance. Code is available at https://github.com/ziplab/Mesa.","url_abs":"https://arxiv.org/abs/2111.11124v3","url_pdf":"https://arxiv.org/pdf/2111.11124v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mesa-a-memory-saving-training-framework-for","repo_url":"https://github.com/zhuang-group/mesa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"mesa-a-memory-saving-training-framework-for","repo_url":"https://github.com/zip-group/mesa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"mesa-a-memory-saving-training-framework-for","repo_url":"https://github.com/ziplab/mesa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2111.11124","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.11124"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zip-group/mesa","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhuang-group/mesa","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ziplab/mesa","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"724a383048928079","entry":"SyncBatchNorm_backward","repo":"zhuang-group/mesa","repo_kind":"official","path":"mesa/custom_bn.py","file_url":"https://github.com/zhuang-group/mesa/blob/HEAD/mesa/custom_bn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"724a383048928079"}},{"code_sha256_prefix":"11eee0d5bc362c59","entry":"SyncBatchNorm_forward","repo":"zhuang-group/mesa","repo_kind":"official","path":"mesa/custom_bn.py","file_url":"https://github.com/zhuang-group/mesa/blob/HEAD/mesa/custom_bn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"11eee0d5bc362c59"}},{"code_sha256_prefix":"380cb5808156c73f","entry":"bn_pre_forward","repo":"zhuang-group/mesa","repo_kind":"official","path":"mesa/custom_bn.py","file_url":"https://github.com/zhuang-group/mesa/blob/HEAD/mesa/custom_bn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"380cb5808156c73f"}},{"code_sha256_prefix":"782cec8c727976e3","entry":"depack_group","repo":"zhuang-group/mesa","repo_kind":"official","path":"mesa/custom_quant.py","file_url":"https://github.com/zhuang-group/mesa/blob/HEAD/mesa/custom_quant.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"782cec8c727976e3"}},{"code_sha256_prefix":"22072e90dd836a6f","entry":"pack_group","repo":"zhuang-group/mesa","repo_kind":"official","path":"mesa/custom_quant.py","file_url":"https://github.com/zhuang-group/mesa/blob/HEAD/mesa/custom_quant.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"22072e90dd836a6f"}},{"code_sha256_prefix":"7f51844be422b72e","entry":"packbits_padded","repo":"zhuang-group/mesa","repo_kind":"official","path":"mesa/packbit.py","file_url":"https://github.com/zhuang-group/mesa/blob/HEAD/mesa/packbit.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f51844be422b72e"}},{"code_sha256_prefix":"53a86e524b38795e","entry":"read_policy","repo":"zhuang-group/mesa","repo_kind":"official","path":"mesa/policy.py","file_url":"https://github.com/zhuang-group/mesa/blob/HEAD/mesa/policy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"53a86e524b38795e"}},{"code_sha256_prefix":"4f56b09e80aef47e","entry":"unpackbits_padded","repo":"zhuang-group/mesa","repo_kind":"official","path":"mesa/packbit.py","file_url":"https://github.com/zhuang-group/mesa/blob/HEAD/mesa/packbit.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4f56b09e80aef47e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}