{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/perfectly-balanced-improving-transfer-and","title":"Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learning","arxiv_id":"2204.07596","date":"2022-04-15","proceeding":null,"authors":["Mayee F. Chen","Daniel Y. Fu","Avanika Narayan","Michael Zhang","Zhao Song","Kayvon Fatahalian","Christopher Ré"],"abstract":"An ideal learned representation should display transferability and robustness. Supervised contrastive learning (SupCon) is a promising method for training accurate models, but produces representations that do not capture these properties due to class collapse -- when all points in a class map to the same representation. Recent work suggests that \"spreading out\" these representations improves them, but the precise mechanism is poorly understood. We argue that creating spread alone is insufficient for better representations, since spread is invariant to permutations within classes. Instead, both the correct degree of spread and a mechanism for breaking this invariance are necessary. We first prove that adding a weighted class-conditional InfoNCE loss to SupCon controls the degree of spread. Next, we study three mechanisms to break permutation invariance: using a constrained encoder, adding a class-conditional autoencoder, and using data augmentation. We show that the latter two encourage clustering of latent subclasses under more realistic conditions than the former. Using these insights, we show that adding a properly-weighted class-conditional InfoNCE loss and a class-conditional autoencoder to SupCon achieves 11.1 points of lift on coarse-to-fine transfer across 5 standard datasets and 4.7 points on worst-group robustness on 3 datasets, setting state-of-the-art on CelebA by 11.5 points.","url_abs":"https://arxiv.org/abs/2204.07596v2","url_pdf":"https://arxiv.org/pdf/2204.07596v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"perfectly-balanced-improving-transfer-and","repo_url":"https://github.com/HazyResearch/thanos-code","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"data-augmentation","task_name":"Data Augmentation"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"infonce","method_name":"InfoNCE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2204.07596","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.07596"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HazyResearch/thanos-code","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4251f968142da139","entry":"ce_loss","repo":"HazyResearch/thanos-code","repo_kind":"official","path":"unagi/tasks/loss_modules.py","file_url":"https://github.com/HazyResearch/thanos-code/blob/HEAD/unagi/tasks/loss_modules.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4251f968142da139"}},{"code_sha256_prefix":"b0a00d47dac2d2f4","entry":"is_list","repo":"HazyResearch/thanos-code","repo_kind":"official","path":"unagi/datasets/base_dataset.py","file_url":"https://github.com/HazyResearch/thanos-code/blob/HEAD/unagi/datasets/base_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b0a00d47dac2d2f4"}},{"code_sha256_prefix":"7cc0adbd026ba6e9","entry":"mask_loss","repo":"HazyResearch/thanos-code","repo_kind":"official","path":"unagi/tasks/loss_modules.py","file_url":"https://github.com/HazyResearch/thanos-code/blob/HEAD/unagi/tasks/loss_modules.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7cc0adbd026ba6e9"}},{"code_sha256_prefix":"d654687ffa322293","entry":"multiclass_classification","repo":"HazyResearch/thanos-code","repo_kind":"official","path":"unagi/tasks/output_layer_modules.py","file_url":"https://github.com/HazyResearch/thanos-code/blob/HEAD/unagi/tasks/output_layer_modules.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d654687ffa322293"}},{"code_sha256_prefix":"13c311b0692c7407","entry":"multilabel_classification","repo":"HazyResearch/thanos-code","repo_kind":"official","path":"unagi/tasks/output_layer_modules.py","file_url":"https://github.com/HazyResearch/thanos-code/blob/HEAD/unagi/tasks/output_layer_modules.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"13c311b0692c7407"}},{"code_sha256_prefix":"9c5cbb4a3040b938","entry":"sce_loss","repo":"HazyResearch/thanos-code","repo_kind":"official","path":"unagi/tasks/loss_modules.py","file_url":"https://github.com/HazyResearch/thanos-code/blob/HEAD/unagi/tasks/loss_modules.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9c5cbb4a3040b938"}},{"code_sha256_prefix":"0d0f48549ba480fa","entry":"sparse2coarse","repo":"HazyResearch/thanos-code","repo_kind":"official","path":"unagi/datasets/tiny_imagenet/utils.py","file_url":"https://github.com/HazyResearch/thanos-code/blob/HEAD/unagi/datasets/tiny_imagenet/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0d0f48549ba480fa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}