{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/memory-efficient-implementation-of-densenets","title":"Memory-Efficient Implementation of DenseNets","arxiv_id":"1707.06990","date":"2017-07-21","proceeding":null,"authors":["Geoff Pleiss","Danlu Chen","Gao Huang","Tongcheng Li","Laurens van der Maaten","Kilian Q. Weinberger"],"abstract":"The DenseNet architecture is highly computationally efficient as a result of\nfeature reuse. However, a naive DenseNet implementation can require a\nsignificant amount of GPU memory: If not properly managed, pre-activation batch\nnormalization and contiguous convolution operations can produce feature maps\nthat grow quadratically with network depth. In this technical report, we\nintroduce strategies to reduce the memory consumption of DenseNets during\ntraining. By strategically using shared memory allocations, we reduce the\nmemory cost for storing feature maps from quadratic to linear. Without the GPU\nmemory bottleneck, it is now possible to train extremely deep DenseNets.\nNetworks with 14M parameters can be trained on a single GPU, up from 4M. A\n264-layer DenseNet (73M parameters), which previously would have been\ninfeasible to train, can now be trained on a single workstation with 8 NVIDIA\nTesla M40 GPUs. On the ImageNet ILSVRC classification dataset, this large\nDenseNet obtains a state-of-the-art single-crop top-1 error of 20.26%.","url_abs":"http://arxiv.org/abs/1707.06990v1","url_pdf":"http://arxiv.org/pdf/1707.06990v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"memory-efficient-implementation-of-densenets","repo_url":"https://github.com/gpleiss/efficient_densenet_pytorch","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"memory-efficient-implementation-of-densenets","repo_url":"https://github.com/facebookresearch/ResNeXt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"torch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"memory-efficient-implementation-of-densenets","repo_url":"https://github.com/joeyearsley/efficient_densenet_tensorflow","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"memory-efficient-implementation-of-densenets","repo_url":"https://github.com/npielawski/pytorch_tiramisu","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"memory-efficient-implementation-of-densenets","repo_url":"https://github.com/zhourunlong/densenet-ag_news-nlp","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"memory-efficient-implementation-of-densenets","repo_url":"https://github.com/code-implementation1/Code2/tree/main/3D_DenseNet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":null,"task_name":"GPU"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"concatenated-skip-connection","method_name":"Concatenated Skip Connection"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-block","method_name":"Dense Block"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1707.06990","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}