{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sharp-learning-bounds-for-contrastive","title":"On the Surrogate Gap between Contrastive and Supervised Losses","arxiv_id":"2110.02501","date":"2021-10-06","proceeding":null,"authors":["Han Bao","Yoshihiro Nagano","Kento Nozawa"],"abstract":"Contrastive representation learning encourages data representation to make semantically similar pairs closer than randomly drawn negative samples, which has been successful in various domains such as vision, language, and graphs. Recent theoretical studies have attempted to explain the benefit of the large negative sample size by upper-bounding the downstream classification loss with the contrastive loss. However, the previous surrogate bounds have two drawbacks: they are only legitimate for a limited range of negative sample sizes and prohibitively large even within that range. Due to these drawbacks, there still does not exist a consensus on how negative sample size theoretically correlates with downstream classification performance. Following the simplified setting where positive pairs are drawn from the true distribution (not generated by data augmentation; as supposed in previous studies), this study establishes surrogate upper and lower bounds for the downstream classification loss for all negative sample sizes that best explain the empirical observations on the negative sample size in the earlier studies. Our bounds suggest that the contrastive loss can be viewed as a surrogate objective of the downstream loss and larger negative sample sizes improve downstream classification because the surrogate gap between contrastive and supervised losses decays. We verify that our theory is consistent with experiments on synthetic, vision, and language datasets.","url_abs":"https://arxiv.org/abs/2110.02501v2","url_pdf":"https://arxiv.org/pdf/2110.02501v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sharp-learning-bounds-for-contrastive","repo_url":"https://github.com/nzw0301/gap-contrastive-and-supervised-losses","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2110.02501","atlas_url":"https://app.syntology.ai/?focus=2110.02501","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.02501"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nzw0301/gap-contrastive-and-supervised-losses","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"313c0a68ab564c5a","entry":"convert_dataset_to_contrastive_dataset","repo":"nzw0301/gap-contrastive-and-supervised-losses","repo_kind":"official","path":"code/src/datasets/contrastive.py","file_url":"https://github.com/nzw0301/gap-contrastive-and-supervised-losses/blob/HEAD/code/src/datasets/contrastive.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"313c0a68ab564c5a"}},{"code_sha256_prefix":"d90bbebfcdbd36a4","entry":"exclude_from_wt_decay","repo":"nzw0301/gap-contrastive-and-supervised-losses","repo_kind":"official","path":"code/vision_contrastive.py","file_url":"https://github.com/nzw0301/gap-contrastive-and-supervised-losses/blob/HEAD/code/vision_contrastive.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d90bbebfcdbd36a4"}},{"code_sha256_prefix":"b8e5ac2c9d6cec4f","entry":"generate_data","repo":"nzw0301/gap-contrastive-and-supervised-losses","repo_kind":"official","path":"code/toy_contrastive.py","file_url":"https://github.com/nzw0301/gap-contrastive-and-supervised-losses/blob/HEAD/code/toy_contrastive.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b8e5ac2c9d6cec4f"}},{"code_sha256_prefix":"9cda64ef2dc34fb4","entry":"rotate","repo":"nzw0301/gap-contrastive-and-supervised-losses","repo_kind":"official","path":"code/toy_contrastive.py","file_url":"https://github.com/nzw0301/gap-contrastive-and-supervised-losses/blob/HEAD/code/toy_contrastive.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9cda64ef2dc34fb4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}