{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ovsw-overcoming-silent-weights-for-accurate","title":"OvSW: Overcoming Silent Weights for Accurate Binary Neural Networks","arxiv_id":"2407.05257","date":"2024-07-07","proceeding":null,"authors":["Jingyang Xiang","Zuohui Chen","Siqi Li","Qing Wu","Yong liu"],"abstract":"Binary Neural Networks~(BNNs) have been proven to be highly effective for deploying deep neural networks on mobile and embedded platforms. Most existing works focus on minimizing quantization errors, improving representation ability, or designing gradient approximations to alleviate gradient mismatch in BNNs, while leaving the weight sign flipping, a critical factor for achieving powerful BNNs, untouched. In this paper, we investigate the efficiency of weight sign updates in BNNs. We observe that, for vanilla BNNs, over 50\\% of the weights remain their signs unchanged during training, and these weights are not only distributed at the tails of the weight distribution but also universally present in the vicinity of zero. We refer to these weights as ``silent weights'', which slow down convergence and lead to a significant accuracy degradation. Theoretically, we reveal this is due to the independence of the BNNs gradient from the latent weight distribution. To address the issue, we propose Overcome Silent Weights~(OvSW). OvSW first employs Adaptive Gradient Scaling~(AGS) to establish a relationship between the gradient and the latent weight distribution, thereby improving the overall efficiency of weight sign updates. Additionally, we design Silence Awareness Decaying~(SAD) to automatically identify ``silent weights'' by tracking weight flipping state, and apply an additional penalty to ``silent weights'' to facilitate their flipping. By efficiently updating weight signs, our method achieves faster convergence and state-of-the-art performance on CIFAR10 and ImageNet1K dataset with various architectures. For example, OvSW obtains 61.6\\% and 65.5\\% top-1 accuracy on the ImageNet1K using binarized ResNet18 and ResNet34 architecture respectively. Codes are available at \\url{https://github.com/JingyangXiang/OvSW}.","url_abs":"https://arxiv.org/abs/2407.05257v1","url_pdf":"https://arxiv.org/pdf/2407.05257v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ovsw-overcoming-silent-weights-for-accurate","repo_url":"https://github.com/JingyangXiang/OvSW","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2407.05257","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.05257"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/JingyangXiang/OvSW","reach":null}],"summary":{"ran":13},"by_repo_kind":{"official":{"samples":13,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"102338457597f4fc","entry":"ActBinaryBase","repo":"JingyangXiang/OvSW","repo_kind":"official","path":"model/modules/ovsw.py","file_url":"https://github.com/JingyangXiang/OvSW/blob/HEAD/model/modules/ovsw.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"102338457597f4fc"}},{"code_sha256_prefix":"5d6f1964a23e443c","entry":"BinaryAct","repo":"JingyangXiang/OvSW","repo_kind":"official","path":"model/modules/ovsw.py","file_url":"https://github.com/JingyangXiang/OvSW/blob/HEAD/model/modules/ovsw.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5d6f1964a23e443c"}},{"code_sha256_prefix":"c06fa39d74a91cf1","entry":"BinaryQuantizedEstimator","repo":"JingyangXiang/OvSW","repo_kind":"official","path":"model/modules/ovsw.py","file_url":"https://github.com/JingyangXiang/OvSW/blob/HEAD/model/modules/ovsw.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c06fa39d74a91cf1"}},{"code_sha256_prefix":"060589672b3411b2","entry":"HardTanhSTE","repo":"JingyangXiang/OvSW","repo_kind":"official","path":"model/modules/ovsw.py","file_url":"https://github.com/JingyangXiang/OvSW/blob/HEAD/model/modules/ovsw.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"060589672b3411b2"}},{"code_sha256_prefix":"96b7cbb0ca8ea100","entry":"HardTanhSTEV2","repo":"JingyangXiang/OvSW","repo_kind":"official","path":"model/modules/ovsw.py","file_url":"https://github.com/JingyangXiang/OvSW/blob/HEAD/model/modules/ovsw.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"96b7cbb0ca8ea100"}},{"code_sha256_prefix":"10f46c53bcdb26bf","entry":"HtanhSTEThroughEstimator","repo":"JingyangXiang/OvSW","repo_kind":"official","path":"model/modules/ovsw.py","file_url":"https://github.com/JingyangXiang/OvSW/blob/HEAD/model/modules/ovsw.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"10f46c53bcdb26bf"}},{"code_sha256_prefix":"d1f9b2090cb0cc0d","entry":"HtanhSTEThroughEstimatorV2","repo":"JingyangXiang/OvSW","repo_kind":"official","path":"model/modules/ovsw.py","file_url":"https://github.com/JingyangXiang/OvSW/blob/HEAD/model/modules/ovsw.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d1f9b2090cb0cc0d"}},{"code_sha256_prefix":"aefaf2f670da16b2","entry":"OvSWConv2d","repo":"JingyangXiang/OvSW","repo_kind":"official","path":"model/modules/ovsw.py","file_url":"https://github.com/JingyangXiang/OvSW/blob/HEAD/model/modules/ovsw.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aefaf2f670da16b2"}},{"code_sha256_prefix":"f19aa2f29f78a567","entry":"STEWeight","repo":"JingyangXiang/OvSW","repo_kind":"official","path":"model/modules/ovsw.py","file_url":"https://github.com/JingyangXiang/OvSW/blob/HEAD/model/modules/ovsw.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f19aa2f29f78a567"}},{"code_sha256_prefix":"445a8f06a606f28f","entry":"STEWeightV2","repo":"JingyangXiang/OvSW","repo_kind":"official","path":"model/modules/ovsw.py","file_url":"https://github.com/JingyangXiang/OvSW/blob/HEAD/model/modules/ovsw.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"445a8f06a606f28f"}},{"code_sha256_prefix":"5273a7709992e68d","entry":"StraightThroughEstimator","repo":"JingyangXiang/OvSW","repo_kind":"official","path":"model/modules/ovsw.py","file_url":"https://github.com/JingyangXiang/OvSW/blob/HEAD/model/modules/ovsw.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5273a7709992e68d"}},{"code_sha256_prefix":"be63621ad326a545","entry":"StraightThroughEstimatorV2","repo":"JingyangXiang/OvSW","repo_kind":"official","path":"model/modules/ovsw.py","file_url":"https://github.com/JingyangXiang/OvSW/blob/HEAD/model/modules/ovsw.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"be63621ad326a545"}},{"code_sha256_prefix":"a1a2651b57e13de9","entry":"WeightBinaryBase","repo":"JingyangXiang/OvSW","repo_kind":"official","path":"model/modules/ovsw.py","file_url":"https://github.com/JingyangXiang/OvSW/blob/HEAD/model/modules/ovsw.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a1a2651b57e13de9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}