{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-accurate-low-bit-deep-neural","title":"Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization","arxiv_id":"1708.01001","date":"2017-08-03","proceeding":null,"authors":["Yinpeng Dong","Renkun Ni","Jianguo Li","Yurong Chen","Jun Zhu","Hang Su"],"abstract":"Low-bit deep neural networks (DNNs) become critical for embedded applications\ndue to their low storage requirement and computing efficiency. However, they\nsuffer much from the non-negligible accuracy drop. This paper proposes the\nstochastic quantization (SQ) algorithm for learning accurate low-bit DNNs. The\nmotivation is due to the following observation. Existing training algorithms\napproximate the real-valued elements/filters with low-bit representation all\ntogether in each iteration. The quantization errors may be small for some\nelements/filters, while are remarkable for others, which lead to inappropriate\ngradient direction during training, and thus bring notable accuracy drop.\nInstead, SQ quantizes a portion of elements/filters to low-bit with a\nstochastic probability inversely proportional to the quantization error, while\nkeeping the other portion unchanged with full-precision. The quantized and\nfull-precision portions are updated with corresponding gradients separately in\neach iteration. The SQ ratio is gradually increased until the whole network is\nquantized. This procedure can greatly compensate the quantization error and\nthus yield better accuracy for low-bit DNNs. Experiments show that SQ can\nconsistently and significantly improve the accuracy for different low-bit DNNs\non various datasets and various network structures.","url_abs":"http://arxiv.org/abs/1708.01001v1","url_pdf":"http://arxiv.org/pdf/1708.01001v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-accurate-low-bit-deep-neural","repo_url":"https://github.com/dongyp13/Stochastic-Quantization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1708.01001","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}