{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/optimal-clipping-and-magnitude-aware","title":"Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training","arxiv_id":"2206.06501","date":"2022-06-13","proceeding":null,"authors":["Charbel Sakr","Steve Dai","Rangharajan Venkatesan","Brian Zimmer","William J. Dally","Brucek Khailany"],"abstract":"Data clipping is crucial in reducing noise in quantization operations and improving the achievable accuracy of quantization-aware training (QAT). Current practices rely on heuristics to set clipping threshold scalars and cannot be shown to be optimal. We propose Optimally Clipped Tensors And Vectors (OCTAV), a recursive algorithm to determine MSE-optimal clipping scalars. Derived from the fast Newton-Raphson method, OCTAV finds optimal clipping scalars on the fly, for every tensor, at every iteration of the QAT routine. Thus, the QAT algorithm is formulated with provably minimum quantization noise at each step. In addition, we reveal limitations in common gradient estimation techniques in QAT and propose magnitude-aware differentiation as a remedy to further improve accuracy. Experimentally, OCTAV-enabled QAT achieves state-of-the-art accuracy on multiple tasks. These include training-from-scratch and retraining ResNets and MobileNets on ImageNet, and Squad fine-tuning using BERT models, where OCTAV-enabled QAT consistently preserves accuracy at low precision (4-to-6-bits). Our results require no modifications to the baseline training recipe, except for the insertion of quantization operations where appropriate.","url_abs":"https://arxiv.org/abs/2206.06501v1","url_pdf":"https://arxiv.org/pdf/2206.06501v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"optimal-clipping-and-magnitude-aware","repo_url":"https://github.com/zavareh1/ClippedQuantFL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2206.06501","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.06501"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zavareh1/ClippedQuantFL","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":9},"by_repo_kind":{"listed":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"608ae6fff31fd3f3","entry":"compute_global_clipping_scalar","repo":"zavareh1/ClippedQuantFL","repo_kind":"listed","path":"model/cifar10DET_quantizedmaxscalar.py","file_url":"https://github.com/zavareh1/ClippedQuantFL/blob/HEAD/model/cifar10DET_quantizedmaxscalar.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"608ae6fff31fd3f3"}},{"code_sha256_prefix":"0cc294de59beb777","entry":"compute_optimal_clipping_scalar","repo":"zavareh1/ClippedQuantFL","repo_kind":"listed","path":"model/cifar10DET_quantizedoctav.py","file_url":"https://github.com/zavareh1/ClippedQuantFL/blob/HEAD/model/cifar10DET_quantizedoctav.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0cc294de59beb777"}},{"code_sha256_prefix":"4518324b5d7bd6f0","entry":"distribute","repo":"zavareh1/ClippedQuantFL","repo_kind":"listed","path":"data.py","file_url":"https://github.com/zavareh1/ClippedQuantFL/blob/HEAD/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4518324b5d7bd6f0"}},{"code_sha256_prefix":"00cb1200d5219763","entry":"quantize","repo":"zavareh1/ClippedQuantFL","repo_kind":"listed","path":"model/cifar10DET_quantizedmaxscalar.py","file_url":"https://github.com/zavareh1/ClippedQuantFL/blob/HEAD/model/cifar10DET_quantizedmaxscalar.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00cb1200d5219763"}},{"code_sha256_prefix":"8b2338dff703aa86","entry":"quantize_weights","repo":"zavareh1/ClippedQuantFL","repo_kind":"listed","path":"model/cifar10DET_quantizedmaxscalar.py","file_url":"https://github.com/zavareh1/ClippedQuantFL/blob/HEAD/model/cifar10DET_quantizedmaxscalar.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b2338dff703aa86"}},{"code_sha256_prefix":"d15db84a08ceff80","entry":"quantize_weights","repo":"zavareh1/ClippedQuantFL","repo_kind":"listed","path":"model/cifar10DET_quantizedoctav.py","file_url":"https://github.com/zavareh1/ClippedQuantFL/blob/HEAD/model/cifar10DET_quantizedoctav.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d15db84a08ceff80"}},{"code_sha256_prefix":"4f8147e48aa2c397","entry":"quantize_weights","repo":"zavareh1/ClippedQuantFL","repo_kind":"listed","path":"model/cifar10STOCH_quantizedmaxscalar.py","file_url":"https://github.com/zavareh1/ClippedQuantFL/blob/HEAD/model/cifar10STOCH_quantizedmaxscalar.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f8147e48aa2c397"}},{"code_sha256_prefix":"fe0240bc2fecf97c","entry":"quantize_weights","repo":"zavareh1/ClippedQuantFL","repo_kind":"listed","path":"model/cifar10STOCH_quantizedoctav.py","file_url":"https://github.com/zavareh1/ClippedQuantFL/blob/HEAD/model/cifar10STOCH_quantizedoctav.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe0240bc2fecf97c"}},{"code_sha256_prefix":"ce8a5a415feadd1f","entry":"read_dir","repo":"zavareh1/ClippedQuantFL","repo_kind":"listed","path":"data.py","file_url":"https://github.com/zavareh1/ClippedQuantFL/blob/HEAD/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce8a5a415feadd1f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}