{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/when-does-contrastive-learning-preserve","title":"When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?","arxiv_id":"2111.01124","date":"2021-11-01","proceeding":"NeurIPS 2021 12","authors":["Lijie Fan","Sijia Liu","Pin-Yu Chen","Gaoyuan Zhang","Chuang Gan"],"abstract":"Contrastive learning (CL) can learn generalizable feature representations and achieve the state-of-the-art performance of downstream tasks by finetuning a linear classifier on top of it. However, as adversarial robustness becomes vital in image classification, it remains unclear whether or not CL is able to preserve robustness to downstream tasks. The main challenge is that in the self-supervised pretraining + supervised finetuning paradigm, adversarial robustness is easily forgotten due to a learning task mismatch from pretraining to finetuning. We call such a challenge 'cross-task robustness transferability'. To address the above problem, in this paper we revisit and advance CL principles through the lens of robustness enhancement. We show that (1) the design of contrastive views matters: High-frequency components of images are beneficial to improving model robustness; (2) Augmenting CL with pseudo-supervision stimulus (e.g., resorting to feature clustering) helps preserve robustness without forgetting. Equipped with our new designs, we propose AdvCL, a novel adversarial contrastive pretraining framework. We show that AdvCL is able to enhance cross-task robustness transferability without loss of model accuracy and finetuning efficiency. With a thorough experimental study, we demonstrate that AdvCL outperforms the state-of-the-art self-supervised robust learning methods across multiple datasets (CIFAR-10, CIFAR-100, and STL-10) and finetuning schemes (linear evaluation and full model finetuning).","url_abs":"https://arxiv.org/abs/2111.01124v1","url_pdf":"https://arxiv.org/pdf/2111.01124v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"when-does-contrastive-learning-preserve","repo_url":"https://github.com/lijiefan/advcl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"when-does-contrastive-learning-preserve","repo_url":"https://github.com/yqy2001/a-infonce","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"},{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"linear-evaluation","task_name":"Linear evaluation"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2111.01124","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.01124"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/LijieFan/AdvCL","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lijiefan/advcl","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yqy2001/a-infonce","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":3,"unverified":9},"by_repo_kind":{"official":{"samples":12,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d9def42110729a85","entry":"conv1x1","repo":"LijieFan/AdvCL","repo_kind":"official","path":"models/resnet_cifar_multibn_ensembleFC.py","file_url":"https://github.com/LijieFan/AdvCL/blob/HEAD/models/resnet_cifar_multibn_ensembleFC.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9def42110729a85"}},{"code_sha256_prefix":"160bb14bd76201b4","entry":"conv3x3","repo":"LijieFan/AdvCL","repo_kind":"official","path":"models/resnet_cifar_multibn_ensembleFC.py","file_url":"https://github.com/LijieFan/AdvCL/blob/HEAD/models/resnet_cifar_multibn_ensembleFC.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"160bb14bd76201b4"}},{"code_sha256_prefix":"8d7ec010c29e813b","entry":"format_time","repo":"LijieFan/AdvCL","repo_kind":"official","path":"utils.py","file_url":"https://github.com/LijieFan/AdvCL/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d7ec010c29e813b"}},{"code_sha256_prefix":"98bb5abdb706fbc6","entry":"ResNet18","repo":"LijieFan/AdvCL","repo_kind":"official","path":"models/resnet_cifar.py","file_url":"https://github.com/LijieFan/AdvCL/blob/HEAD/models/resnet_cifar.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98bb5abdb706fbc6"}},{"code_sha256_prefix":"94f4059c6a99ea7a","entry":"ResNet34","repo":"LijieFan/AdvCL","repo_kind":"official","path":"models/resnet_cifar.py","file_url":"https://github.com/LijieFan/AdvCL/blob/HEAD/models/resnet_cifar.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94f4059c6a99ea7a"}},{"code_sha256_prefix":"b87e9bb2803bbb1c","entry":"ResNet50","repo":"LijieFan/AdvCL","repo_kind":"official","path":"models/resnet_cifar.py","file_url":"https://github.com/LijieFan/AdvCL/blob/HEAD/models/resnet_cifar.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b87e9bb2803bbb1c"}},{"code_sha256_prefix":"07a57c775004c229","entry":"distance","repo":"LijieFan/AdvCL","repo_kind":"official","path":"fr_util.py","file_url":"https://github.com/LijieFan/AdvCL/blob/HEAD/fr_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"07a57c775004c229"}},{"code_sha256_prefix":"c0ee4d6324a9fd1c","entry":"generate_high","repo":"LijieFan/AdvCL","repo_kind":"official","path":"fr_util.py","file_url":"https://github.com/LijieFan/AdvCL/blob/HEAD/fr_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0ee4d6324a9fd1c"}},{"code_sha256_prefix":"36f7a4371ed533f0","entry":"get_mean_and_std","repo":"LijieFan/AdvCL","repo_kind":"official","path":"utils.py","file_url":"https://github.com/LijieFan/AdvCL/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"36f7a4371ed533f0"}},{"code_sha256_prefix":"12d26d3b0fb6ffae","entry":"load_BN_checkpoint","repo":"LijieFan/AdvCL","repo_kind":"official","path":"utils.py","file_url":"https://github.com/LijieFan/AdvCL/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"12d26d3b0fb6ffae"}},{"code_sha256_prefix":"94c2cc0c705e0339","entry":"mask_radial","repo":"LijieFan/AdvCL","repo_kind":"official","path":"fr_util.py","file_url":"https://github.com/LijieFan/AdvCL/blob/HEAD/fr_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94c2cc0c705e0339"}},{"code_sha256_prefix":"28543a18e1279a31","entry":"set_loader","repo":"LijieFan/AdvCL","repo_kind":"official","path":"finetuning_advCL_SLF.py","file_url":"https://github.com/LijieFan/AdvCL/blob/HEAD/finetuning_advCL_SLF.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"28543a18e1279a31"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}