{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/training-deep-nets-with-sublinear-memory-cost","title":"Training Deep Nets with Sublinear Memory Cost","arxiv_id":"1604.06174","date":"2016-04-21","proceeding":null,"authors":["Tianqi Chen","Bing Xu","Chiyuan Zhang","Carlos Guestrin"],"abstract":"We propose a systematic approach to reduce the memory consumption of deep\nneural network training. Specifically, we design an algorithm that costs\nO(sqrt(n)) memory to train a n layer network, with only the computational cost\nof an extra forward pass per mini-batch. As many of the state-of-the-art models\nhit the upper bound of the GPU memory, our algorithm allows deeper and more\ncomplex models to be explored, and helps advance the innovations in deep\nlearning research. We focus on reducing the memory cost to store the\nintermediate feature maps and gradients during training. Computation graph\nanalysis is used for automatic in-place operation and memory sharing\noptimizations. We show that it is possible to trade computation for memory -\ngiving a more memory efficient training algorithm with a little extra\ncomputation cost. In the extreme case, our analysis also shows that the memory\nconsumption can be reduced to O(log n) with as little as O(n log n) extra cost\nfor forward computation. Our experiments show that we can reduce the memory\ncost of a 1,000-layer deep residual network from 48G to 7G with only 30 percent\nadditional running time cost on ImageNet problems. Similarly, significant\nmemory cost reduction is observed in training complex recurrent neural networks\non very long sequences.","url_abs":"http://arxiv.org/abs/1604.06174v2","url_pdf":"http://arxiv.org/pdf/1604.06174v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"training-deep-nets-with-sublinear-memory-cost","repo_url":"https://github.com/AD1024/TorchCheckpointing","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"training-deep-nets-with-sublinear-memory-cost","repo_url":"https://github.com/AD1024/torch-checkpointing","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"training-deep-nets-with-sublinear-memory-cost","repo_url":"https://github.com/FrankLeeeee/HPCAI-2020-BERT-Submission","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"training-deep-nets-with-sublinear-memory-cost","repo_url":"https://github.com/cybertronai/gradient-checkpointing","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"training-deep-nets-with-sublinear-memory-cost","repo_url":"https://github.com/openai/gradient-checkpointing","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"training-deep-nets-with-sublinear-memory-cost","repo_url":"https://github.com/timmywanttolearn/fintune","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":null,"task_name":"GPU"}],"methods":[{"method_slug":"gradient-checkpointing","method_name":"Gradient Checkpointing"}],"datasets_introduced":[],"methods_introduced":[{"slug":"gradient-checkpointing","name":"Gradient Checkpointing","full_name":"Gradient Checkpointing"}],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1604.06174","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1604.06174"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cybertronai/gradient-checkpointing","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/openai/gradient-checkpointing","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AD1024/torch-checkpointing","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/FrankLeeeee/HPCAI-2020-BERT-Submission","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AD1024/TorchCheckpointing","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/timmywanttolearn/fintune","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"6c3322607cafa0ae","entry":"create_name","repo":"AD1024/torch-checkpointing","repo_kind":"listed","path":"utils/checkpointing.py","file_url":"https://github.com/AD1024/torch-checkpointing/blob/HEAD/utils/checkpointing.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6c3322607cafa0ae"}},{"code_sha256_prefix":"f79b25e8e9f6d799","entry":"get_shape","repo":"AD1024/torch-checkpointing","repo_kind":"listed","path":"utils/checkpointing.py","file_url":"https://github.com/AD1024/torch-checkpointing/blob/HEAD/utils/checkpointing.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f79b25e8e9f6d799"}},{"code_sha256_prefix":"c4d2c5b1e6baa881","entry":"get_value","repo":"AD1024/torch-checkpointing","repo_kind":"listed","path":"utils/checkpointing.py","file_url":"https://github.com/AD1024/torch-checkpointing/blob/HEAD/utils/checkpointing.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c4d2c5b1e6baa881"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}