{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/revisiting-bfloat16-training","title":"Revisiting BFloat16 Training","arxiv_id":"2010.06192","date":"2020-10-13","proceeding":null,"authors":["Pedram Zamirai","Jian Zhang","Christopher R. Aberger","Christopher De Sa"],"abstract":"State-of-the-art generic low-precision training algorithms use a mix of 16-bit and 32-bit precision, creating the folklore that 16-bit hardware compute units alone are not enough to maximize model accuracy. As a result, deep learning accelerators are forced to support both 16-bit and 32-bit floating-point units (FPUs), which is more costly than only using 16-bit FPUs for hardware design. We ask: can we train deep learning models only with 16-bit floating-point units, while still matching the model accuracy attained by 32-bit training? Towards this end, we study 16-bit-FPU training on the widely adopted BFloat16 unit. While these units conventionally use nearest rounding to cast output to 16-bit precision, we show that nearest rounding for model weight updates often cancels small updates, which degrades the convergence and model accuracy. Motivated by this, we study two simple techniques well-established in numerical analysis, stochastic rounding and Kahan summation, to remedy the model accuracy degradation in 16-bit-FPU training. We demonstrate that these two techniques can enable up to 7% absolute validation accuracy gain in 16-bit-FPU training. This leads to 0.1% lower to 0.2% higher validation accuracy compared to 32-bit training across seven deep learning applications.","url_abs":"https://arxiv.org/abs/2010.06192v2","url_pdf":"https://arxiv.org/pdf/2010.06192v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"revisiting-bfloat16-training","repo_url":"https://github.com/eole-nlp/eole","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2010.06192","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.06192"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eole-nlp/eole","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"unverified":7},"by_repo_kind":{"listed":{"samples":8,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e62e48a00ed06f63","entry":"pad_at_dim","repo":"eole-nlp/eole","repo_kind":"listed","path":"eole/modules/alibi_position_bias.py","file_url":"https://github.com/eole-nlp/eole/blob/HEAD/eole/modules/alibi_position_bias.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e62e48a00ed06f63"}},{"code_sha256_prefix":"0a7d3b2dea77a6c8","entry":"extract_chunks","repo":"eole-nlp/eole","repo_kind":"listed","path":"apps/eole-translator.py","file_url":"https://github.com/eole-nlp/eole/blob/HEAD/apps/eole-translator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a7d3b2dea77a6c8"}},{"code_sha256_prefix":"5482b0753733ae86","entry":"get_base_url","repo":"eole-nlp/eole","repo_kind":"listed","path":"apps/eole-translator.py","file_url":"https://github.com/eole-nlp/eole/blob/HEAD/apps/eole-translator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5482b0753733ae86"}},{"code_sha256_prefix":"20b14ae6f5d3a549","entry":"hf_cache_dir","repo":"eole-nlp/eole","repo_kind":"listed","path":"eole/models/hf_resolver.py","file_url":"https://github.com/eole-nlp/eole/blob/HEAD/eole/models/hf_resolver.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20b14ae6f5d3a549"}},{"code_sha256_prefix":"9ff19f37657a0682","entry":"is_hf_model_id","repo":"eole-nlp/eole","repo_kind":"listed","path":"eole/models/hf_resolver.py","file_url":"https://github.com/eole-nlp/eole/blob/HEAD/eole/models/hf_resolver.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ff19f37657a0682"}},{"code_sha256_prefix":"4ed31e70485394dc","entry":"rebuild_text","repo":"eole-nlp/eole","repo_kind":"listed","path":"apps/eole-translator.py","file_url":"https://github.com/eole-nlp/eole/blob/HEAD/apps/eole-translator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ed31e70485394dc"}},{"code_sha256_prefix":"5e8fa1dfb99981ed","entry":"replace_awq_linear","repo":"eole-nlp/eole","repo_kind":"listed","path":"eole/modules/awq_linear.py","file_url":"https://github.com/eole-nlp/eole/blob/HEAD/eole/modules/awq_linear.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e8fa1dfb99981ed"}},{"code_sha256_prefix":"21f09e367ce1ec22","entry":"replace_bnb_linear","repo":"eole-nlp/eole","repo_kind":"listed","path":"eole/modules/bnb_linear.py","file_url":"https://github.com/eole-nlp/eole/blob/HEAD/eole/modules/bnb_linear.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21f09e367ce1ec22"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}