{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/s2-bnn-bridging-the-gap-between-self","title":"S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration","arxiv_id":"2102.08946","date":"2021-02-17","proceeding":"CVPR 2021 1","authors":["Zhiqiang Shen","Zechun Liu","Jie Qin","Lei Huang","Kwang-Ting Cheng","Marios Savvides"],"abstract":"Previous studies dominantly target at self-supervised learning on real-valued networks and have achieved many promising results. However, on the more challenging binary neural networks (BNNs), this task has not yet been fully explored in the community. In this paper, we focus on this more difficult scenario: learning networks where both weights and activations are binary, meanwhile, without any human annotated labels. We observe that the commonly used contrastive objective is not satisfying on BNNs for competitive accuracy, since the backbone network contains relatively limited capacity and representation ability. Hence instead of directly applying existing self-supervised methods, which cause a severe decline in performance, we present a novel guided learning paradigm from real-valued to distill binary networks on the final prediction distribution, to minimize the loss and obtain desirable accuracy. Our proposed method can boost the simple contrastive learning baseline by an absolute gain of 5.5~15% on BNNs. We further reveal that it is difficult for BNNs to recover the similar predictive distributions as real-valued models when training without labels. Thus, how to calibrate them is key to address the degradation in performance. Extensive experiments are conducted on the large-scale ImageNet and downstream datasets. Our method achieves substantial improvement over the simple contrastive learning baseline, and is even comparable to many mainstream supervised BNN methods. Code is available at https://github.com/szq0214/S2-BNN.","url_abs":"https://arxiv.org/abs/2102.08946v2","url_pdf":"https://arxiv.org/pdf/2102.08946v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"s2-bnn-bridging-the-gap-between-self","repo_url":"https://github.com/szq0214/S2-BNN","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2102.08946","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2102.08946"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/szq0214/S2-BNN","reach":null}],"summary":{"ran":4,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"ab6e88abf32692de","entry":"HardBinaryConv","repo":"szq0214/S2-BNN","repo_kind":"official","path":"Contrastive+Distillation/step2/ReActNet.py","file_url":"https://github.com/szq0214/S2-BNN/blob/HEAD/Contrastive%2BDistillation/step2/ReActNet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab6e88abf32692de"}},{"code_sha256_prefix":"605b8a3cb1419388","entry":"LearnableBias","repo":"szq0214/S2-BNN","repo_kind":"official","path":"Contrastive+Distillation/step2/ReActNet.py","file_url":"https://github.com/szq0214/S2-BNN/blob/HEAD/Contrastive%2BDistillation/step2/ReActNet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"605b8a3cb1419388"}},{"code_sha256_prefix":"ef31fea12fd894ec","entry":"ReActNetBlock","repo":"szq0214/S2-BNN","repo_kind":"official","path":"Contrastive+Distillation/step2/ReActNet.py","file_url":"https://github.com/szq0214/S2-BNN/blob/HEAD/Contrastive%2BDistillation/step2/ReActNet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ef31fea12fd894ec"}},{"code_sha256_prefix":"2be1d1898e0e4f99","entry":"binaryconv1x1","repo":"szq0214/S2-BNN","repo_kind":"official","path":"Contrastive+Distillation/step2/ReActNet.py","file_url":"https://github.com/szq0214/S2-BNN/blob/HEAD/Contrastive%2BDistillation/step2/ReActNet.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2be1d1898e0e4f99"}},{"code_sha256_prefix":"a9aa7207ec783915","entry":"binaryconv3x3","repo":"szq0214/S2-BNN","repo_kind":"official","path":"Contrastive+Distillation/step2/ReActNet.py","file_url":"https://github.com/szq0214/S2-BNN/blob/HEAD/Contrastive%2BDistillation/step2/ReActNet.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a9aa7207ec783915"}},{"code_sha256_prefix":"0fca2ac38b728458","entry":"firstconv3x3","repo":"szq0214/S2-BNN","repo_kind":"official","path":"Contrastive+Distillation/step2/ReActNet.py","file_url":"https://github.com/szq0214/S2-BNN/blob/HEAD/Contrastive%2BDistillation/step2/ReActNet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0fca2ac38b728458"}},{"code_sha256_prefix":"0d330eb7f5eaf415","entry":"ReActNet","repo":"szq0214/S2-BNN","repo_kind":"official","path":"Contrastive+Distillation/step2/ReActNet.py","file_url":"https://github.com/szq0214/S2-BNN/blob/HEAD/Contrastive%2BDistillation/step2/ReActNet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0d330eb7f5eaf415"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}