{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scalable-methods-for-8-bit-training-of-neural","title":"Scalable Methods for 8-bit Training of Neural Networks","arxiv_id":"1805.11046","date":"2018-05-25","proceeding":"NeurIPS 2018 12","authors":["Ron Banner","Itay Hubara","Elad Hoffer","Daniel Soudry"],"abstract":"Quantized Neural Networks (QNNs) are often used to improve network efficiency\nduring the inference phase, i.e. after the network has been trained. Extensive\nresearch in the field suggests many different quantization schemes. Still, the\nnumber of bits required, as well as the best quantization scheme, are yet\nunknown. Our theoretical analysis suggests that most of the training process is\nrobust to substantial precision reduction, and points to only a few specific\noperations that require higher precision. Armed with this knowledge, we\nquantize the model parameters, activations and layer gradients to 8-bit,\nleaving at a higher precision only the final step in the computation of the\nweight gradients. Additionally, as QNNs require batch-normalization to be\ntrained at high precision, we introduce Range Batch-Normalization (BN) which\nhas significantly higher tolerance to quantization noise and improved\ncomputational complexity. Our simulations show that Range BN is equivalent to\nthe traditional batch norm if a precise scale adjustment, which can be\napproximated analytically, is applied. To the best of the authors' knowledge,\nthis work is the first to quantize the weights, activations, as well as a\nsubstantial volume of the gradients stream, in all layers (including batch\nnormalization) to 8-bit while showing state-of-the-art results over the\nImageNet-1K dataset.","url_abs":"http://arxiv.org/abs/1805.11046v3","url_pdf":"http://arxiv.org/pdf/1805.11046v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scalable-methods-for-8-bit-training-of-neural","repo_url":"https://github.com/eladhoffer/quantized.pytorch","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"scalable-methods-for-8-bit-training-of-neural","repo_url":"https://github.com/eladhoffer/convNet.pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"scalable-methods-for-8-bit-training-of-neural","repo_url":"https://github.com/vaapopescu/gradient-pruning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1805.11046","atlas_url":"https://app.syntology.ai/?focus=1805.11046","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1805.11046"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eladhoffer/convNet.pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eladhoffer/quantized.pytorch","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vaapopescu/gradient-pruning","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"unverified":11},"by_repo_kind":{"official":{"samples":13,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fac5364e2f53c6db","entry":"conv3x3","repo":"eladhoffer/quantized.pytorch","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/eladhoffer/quantized.pytorch/blob/HEAD/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fac5364e2f53c6db"}},{"code_sha256_prefix":"bc359191fdd6e179","entry":"conv3x3","repo":"eladhoffer/quantized.pytorch","repo_kind":"official","path":"models/resnext.py","file_url":"https://github.com/eladhoffer/quantized.pytorch/blob/HEAD/models/resnext.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc359191fdd6e179"}},{"code_sha256_prefix":"2aae5266d40678b7","entry":"conv3x3","repo":"eladhoffer/quantized.pytorch","repo_kind":"official","path":"models/resnet_quantized.py","file_url":"https://github.com/eladhoffer/quantized.pytorch/blob/HEAD/models/resnet_quantized.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2aae5266d40678b7"}},{"code_sha256_prefix":"3d9a75a675636b11","entry":"conv3x3","repo":"eladhoffer/quantized.pytorch","repo_kind":"official","path":"models/resnet_quantized_float_bn.py","file_url":"https://github.com/eladhoffer/quantized.pytorch/blob/HEAD/models/resnet_quantized_float_bn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d9a75a675636b11"}},{"code_sha256_prefix":"e6c8bce847358ab5","entry":"conv3x3","repo":"eladhoffer/quantized.pytorch","repo_kind":"official","path":"models/resnet_bwn.py","file_url":"https://github.com/eladhoffer/quantized.pytorch/blob/HEAD/models/resnet_bwn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e6c8bce847358ab5"}},{"code_sha256_prefix":"36d164db20c45f6d","entry":"conv_bn","repo":"eladhoffer/quantized.pytorch","repo_kind":"official","path":"models/inception_resnet_v2.py","file_url":"https://github.com/eladhoffer/quantized.pytorch/blob/HEAD/models/inception_resnet_v2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"36d164db20c45f6d"}},{"code_sha256_prefix":"c852fbd4f3361928","entry":"conv_bn","repo":"eladhoffer/quantized.pytorch","repo_kind":"official","path":"models/inception_v2.py","file_url":"https://github.com/eladhoffer/quantized.pytorch/blob/HEAD/models/inception_v2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c852fbd4f3361928"}},{"code_sha256_prefix":"479e6512d18a4cd5","entry":"depBatchNorm2d","repo":"eladhoffer/quantized.pytorch","repo_kind":"official","path":"models/resnext.py","file_url":"https://github.com/eladhoffer/quantized.pytorch/blob/HEAD/models/resnext.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"479e6512d18a4cd5"}},{"code_sha256_prefix":"cde16b7e9575ccb9","entry":"get_dataset","repo":"eladhoffer/quantized.pytorch","repo_kind":"official","path":"data.py","file_url":"https://github.com/eladhoffer/quantized.pytorch/blob/HEAD/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cde16b7e9575ccb9"}},{"code_sha256_prefix":"76d3620502ea711f","entry":"inception_preproccess","repo":"eladhoffer/quantized.pytorch","repo_kind":"official","path":"preprocess.py","file_url":"https://github.com/eladhoffer/quantized.pytorch/blob/HEAD/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"76d3620502ea711f"}},{"code_sha256_prefix":"f0e3c7e07f4ec4a8","entry":"nearby_int","repo":"eladhoffer/quantized.pytorch","repo_kind":"official","path":"models/mobilenet_quantized.py","file_url":"https://github.com/eladhoffer/quantized.pytorch/blob/HEAD/models/mobilenet_quantized.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f0e3c7e07f4ec4a8"}},{"code_sha256_prefix":"8050fdcf3ccf70b9","entry":"pad_random_crop","repo":"eladhoffer/quantized.pytorch","repo_kind":"official","path":"preprocess.py","file_url":"https://github.com/eladhoffer/quantized.pytorch/blob/HEAD/preprocess.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8050fdcf3ccf70b9"}},{"code_sha256_prefix":"a90e11536c336bf6","entry":"scale_crop","repo":"eladhoffer/quantized.pytorch","repo_kind":"official","path":"preprocess.py","file_url":"https://github.com/eladhoffer/quantized.pytorch/blob/HEAD/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a90e11536c336bf6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}