{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/in-place-activated-batchnorm-for-memory","title":"In-Place Activated BatchNorm for Memory-Optimized Training of DNNs","arxiv_id":"1712.02616","date":"2017-12-07","proceeding":"CVPR 2018 6","authors":["Samuel Rota Bulò","Lorenzo Porzi","Peter Kontschieder"],"abstract":"In this work we present In-Place Activated Batch Normalization (InPlace-ABN)\n- a novel approach to drastically reduce the training memory footprint of\nmodern deep neural networks in a computationally efficient way. Our solution\nsubstitutes the conventionally used succession of BatchNorm + Activation layers\nwith a single plugin layer, hence avoiding invasive framework surgery while\nproviding straightforward applicability for existing deep learning frameworks.\nWe obtain memory savings of up to 50% by dropping intermediate results and by\nrecovering required information during the backward pass through the inversion\nof stored forward results, with only minor increase (0.8-2%) in computation\ntime. Also, we demonstrate how frequently used checkpointing approaches can be\nmade computationally as efficient as InPlace-ABN. In our experiments on image\nclassification, we demonstrate on-par results on ImageNet-1k with\nstate-of-the-art approaches. On the memory-demanding task of semantic\nsegmentation, we report results for COCO-Stuff, Cityscapes and Mapillary\nVistas, obtaining new state-of-the-art results on the latter without additional\ntraining data but in a single-scale and -model scenario. Code can be found at\nhttps://github.com/mapillary/inplace_abn .","url_abs":"http://arxiv.org/abs/1712.02616v3","url_pdf":"http://arxiv.org/pdf/1712.02616v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"in-place-activated-batchnorm-for-memory","repo_url":"https://github.com/mapillary/inplace_abn","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"in-place-activated-batchnorm-for-memory","repo_url":"https://github.com/ternaus/TernausNetV2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"inplace-abn","method_name":"InPlace-ABN"}],"datasets_introduced":[],"methods_introduced":[{"slug":"inplace-abn","name":"InPlace-ABN","full_name":"In-Place Activated Batch Normalization"}],"results":[{"leaderboard":"/sota/semantic-segmentation-on-kitti-semantic","task":"Semantic Segmentation","dataset":"KITTI Semantic Segmentation","model":"MapillaryAI","rank_in_archive_order":3,"of":7,"metrics":{"Mean IoU (class)":"69.56"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1712.02616","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}