{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-zipml-framework-for-training-models-with","title":"The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning","arxiv_id":"1611.05402","date":"2016-11-16","proceeding":null,"authors":["Hantian Zhang","Jerry Li","Kaan Kara","Dan Alistarh","Ji Liu","Ce Zhang"],"abstract":"Recently there has been significant interest in training machine-learning\nmodels at low precision: by reducing precision, one can reduce computation and\ncommunication by one order of magnitude. We examine training at reduced\nprecision, both from a theoretical and practical perspective, and ask: is it\npossible to train models at end-to-end low precision with provable guarantees?\nCan this lead to consistent order-of-magnitude speedups? We present a framework\ncalled ZipML to answer these questions. For linear models, the answer is yes.\nWe develop a simple framework based on one simple but novel strategy called\ndouble sampling. Our framework is able to execute training at low precision\nwith no bias, guaranteeing convergence, whereas naive quantization would\nintroduce significant bias. We validate our framework across a range of\napplications, and show that it enables an FPGA prototype that is up to 6.5x\nfaster than an implementation using full 32-bit precision. We further develop a\nvariance-optimal stochastic quantization strategy and show that it can make a\nsignificant difference in a variety of settings. When applied to linear models\ntogether with double sampling, we save up to another 1.7x in data movement\ncompared with uniform quantization. When training deep networks with quantized\nmodels, we achieve higher accuracy than the state-of-the-art XNOR-Net. Finally,\nwe extend our framework through approximation to non-linear models, such as\nSVM. We show that, although using low-precision data induces bias, we can\nappropriately bound and control the bias. We find in practice 8-bit precision\nis often sufficient to converge to the correct solution. Interestingly,\nhowever, in practice we notice that our framework does not always outperform\nthe naive rounding approach. We discuss this negative result in detail.","url_abs":"http://arxiv.org/abs/1611.05402v3","url_pdf":"http://arxiv.org/pdf/1611.05402v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-zipml-framework-for-training-models-with","repo_url":"https://github.com/IST-DASLab/smart-quantizer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1611.05402","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}