{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mixed-precision-training","title":"Mixed Precision Training","arxiv_id":"1710.03740","date":"2017-10-10","proceeding":"ICLR 2018 1","authors":["Paulius Micikevicius","Sharan Narang","Jonah Alben","Gregory Diamos","Erich Elsen","David Garcia","Boris Ginsburg","Michael Houston","Oleksii Kuchaiev","Ganesh Venkatesh","Hao Wu"],"abstract":"Deep neural networks have enabled progress in a wide variety of applications.\nGrowing the size of the neural network typically results in improved accuracy.\nAs model sizes grow, the memory and compute requirements for training these\nmodels also increases. We introduce a technique to train deep neural networks\nusing half precision floating point numbers. In our technique, weights,\nactivations and gradients are stored in IEEE half-precision format.\nHalf-precision floating numbers have limited numerical range compared to\nsingle-precision numbers. We propose two techniques to handle this loss of\ninformation. Firstly, we recommend maintaining a single-precision copy of the\nweights that accumulates the gradients after each optimizer step. This\nsingle-precision copy is rounded to half-precision format during training.\nSecondly, we propose scaling the loss appropriately to handle the loss of\ninformation with half-precision gradients. We demonstrate that this approach\nworks for a wide variety of models including convolution neural networks,\nrecurrent neural networks and generative adversarial networks. This technique\nworks for large scale models with more than 100 million parameters trained on\nlarge datasets. Using this approach, we can reduce the memory consumption of\ndeep learning models by nearly 2x. In future processors, we can also expect a\nsignificant computation speedup using half-precision hardware units.","url_abs":"http://arxiv.org/abs/1710.03740v3","url_pdf":"http://arxiv.org/pdf/1710.03740v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mixed-precision-training","repo_url":"https://github.com/baidu-research/DeepBench","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"mixed-precision-training","repo_url":"https://github.com/NVIDIA/DeepRecommender","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"mixed-precision-training","repo_url":"https://github.com/ajoshiusc/qrsegment","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"mixed-precision-training","repo_url":"https://github.com/deepmind/jmp","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"mixed-precision-training","repo_url":"https://github.com/kaileymonn/Quantized-ENAS-ConvNets","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"mixed-precision-training","repo_url":"https://github.com/milesial/Pytorch-UNet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"mixed-precision-training","repo_url":"https://github.com/ruishengsu/cave_dsa","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"mixed-precision-training","repo_url":"https://github.com/PaddlePaddle/FleetX/blob/develop/examples/resnet/train_fleet_static_amp.py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":null},{"paper_slug":"mixed-precision-training","repo_url":"https://github.com/open-mmlab/mmpose","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"convolution","method_name":"Convolution"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1710.03740","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1710.03740"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/milesial/Pytorch-UNet","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/NVIDIA/DeepRecommender","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kaileymonn/Quantized-ENAS-ConvNets","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/open-mmlab/mmpose","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ruishengsu/cave_dsa","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepmind/jmp","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PaddlePaddle/FleetX/blob/develop/examples/resnet/train_fleet_static_amp.py","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ajoshiusc/qrsegment","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/baidu-research/DeepBench","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":8},"by_repo_kind":{"listed":{"samples":8,"ran":0,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b8a83ff3f54df1bf","entry":"MSEloss","repo":"NVIDIA/DeepRecommender","repo_kind":"listed","path":"reco_encoder/model/model.py","file_url":"https://github.com/NVIDIA/DeepRecommender/blob/HEAD/reco_encoder/model/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b8a83ff3f54df1bf"}},{"code_sha256_prefix":"509cf6cb6046a956","entry":"activation","repo":"NVIDIA/DeepRecommender","repo_kind":"listed","path":"reco_encoder/model/model.py","file_url":"https://github.com/NVIDIA/DeepRecommender/blob/HEAD/reco_encoder/model/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"509cf6cb6046a956"}},{"code_sha256_prefix":"0e1cadfd6ea7a4b9","entry":"all_finite","repo":"deepmind/jmp","repo_kind":"listed","path":"jmp/_src/loss_scale.py","file_url":"https://github.com/deepmind/jmp/blob/HEAD/jmp/_src/loss_scale.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0e1cadfd6ea7a4b9"}},{"code_sha256_prefix":"3fb1fcab480a4e22","entry":"cast_to_full","repo":"deepmind/jmp","repo_kind":"listed","path":"jmp/_src/policy.py","file_url":"https://github.com/deepmind/jmp/blob/HEAD/jmp/_src/policy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3fb1fcab480a4e22"}},{"code_sha256_prefix":"695694e8d73aca88","entry":"cast_to_half","repo":"deepmind/jmp","repo_kind":"listed","path":"jmp/_src/policy.py","file_url":"https://github.com/deepmind/jmp/blob/HEAD/jmp/_src/policy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"695694e8d73aca88"}},{"code_sha256_prefix":"2aea58b608cb15bf","entry":"create_NETFLIX_data_timesplit","repo":"NVIDIA/DeepRecommender","repo_kind":"listed","path":"data_utils/netflix_data_convert.py","file_url":"https://github.com/NVIDIA/DeepRecommender/blob/HEAD/data_utils/netflix_data_convert.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2aea58b608cb15bf"}},{"code_sha256_prefix":"6fcfc8688017b242","entry":"get_policy","repo":"deepmind/jmp","repo_kind":"listed","path":"jmp/_src/policy.py","file_url":"https://github.com/deepmind/jmp/blob/HEAD/jmp/_src/policy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6fcfc8688017b242"}},{"code_sha256_prefix":"6c5df50b2e425ef5","entry":"select_tree","repo":"deepmind/jmp","repo_kind":"listed","path":"jmp/_src/loss_scale.py","file_url":"https://github.com/deepmind/jmp/blob/HEAD/jmp/_src/loss_scale.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6c5df50b2e425ef5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}