{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/memorization-in-self-supervised-learning","title":"Memorization in Self-Supervised Learning Improves Downstream Generalization","arxiv_id":"2401.12233","date":"2024-01-19","proceeding":null,"authors":["Wenhao Wang","Muhammad Ahmad Kaleem","Adam Dziedzic","Michael Backes","Nicolas Papernot","Franziska Boenisch"],"abstract":"Self-supervised learning (SSL) has recently received significant attention due to its ability to train high-performance encoders purely on unlabeled data-often scraped from the internet. This data can still be sensitive and empirical evidence suggests that SSL encoders memorize private information of their training data and can disclose them at inference time. Since existing theoretical definitions of memorization from supervised learning rely on labels, they do not transfer to SSL. To address this gap, we propose SSLMem, a framework for defining memorization within SSL. Our definition compares the difference in alignment of representations for data points and their augmented views returned by both encoders that were trained on these data points and encoders that were not. Through comprehensive empirical analysis on diverse encoder architectures and datasets we highlight that even though SSL relies on large datasets and strong augmentations-both known in supervised learning as regularization techniques that reduce overfitting-still significant fractions of training data points experience high memorization. Through our empirical results, we show that this memorization is essential for encoders to achieve higher generalization performance on different downstream tasks.","url_abs":"https://arxiv.org/abs/2401.12233v3","url_pdf":"https://arxiv.org/pdf/2401.12233v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"memorization-in-self-supervised-learning","repo_url":"https://github.com/sprintml/memorizationssl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"memorization","task_name":"Memorization"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.12233","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.12233"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sprintml/memorizationssl","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9b0d639a8e17734a","entry":"array_gt","repo":"sprintml/memorizationssl","repo_kind":"official","path":"memorization_scores.py","file_url":"https://github.com/sprintml/memorizationssl/blob/HEAD/memorization_scores.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b0d639a8e17734a"}},{"code_sha256_prefix":"c19231378e41dc5a","entry":"bool_flag","repo":"sprintml/memorizationssl","repo_kind":"official","path":"models/DINO/utils.py","file_url":"https://github.com/sprintml/memorizationssl/blob/HEAD/models/DINO/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c19231378e41dc5a"}},{"code_sha256_prefix":"d724ef741e3e3a34","entry":"chng","repo":"sprintml/memorizationssl","repo_kind":"official","path":"memorization_scores.py","file_url":"https://github.com/sprintml/memorizationssl/blob/HEAD/memorization_scores.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d724ef741e3e3a34"}},{"code_sha256_prefix":"2b5489053b222e00","entry":"clip_gradients","repo":"sprintml/memorizationssl","repo_kind":"official","path":"models/DINO/utils.py","file_url":"https://github.com/sprintml/memorizationssl/blob/HEAD/models/DINO/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b5489053b222e00"}},{"code_sha256_prefix":"361a6b24f11fc50a","entry":"cosine_scheduler","repo":"sprintml/memorizationssl","repo_kind":"official","path":"models/DINO/utils.py","file_url":"https://github.com/sprintml/memorizationssl/blob/HEAD/models/DINO/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"361a6b24f11fc50a"}},{"code_sha256_prefix":"fe24b1f5732af5a8","entry":"load_data","repo":"sprintml/memorizationssl","repo_kind":"official","path":"memorization_scores.py","file_url":"https://github.com/sprintml/memorizationssl/blob/HEAD/memorization_scores.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe24b1f5732af5a8"}},{"code_sha256_prefix":"0e027a09795d74d6","entry":"ttest","repo":"sprintml/memorizationssl","repo_kind":"official","path":"Ttest.py","file_url":"https://github.com/sprintml/memorizationssl/blob/HEAD/Ttest.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e027a09795d74d6"}},{"code_sha256_prefix":"e12dc0e9953ba2ea","entry":"test","repo":"sprintml/memorizationssl","repo_kind":"official","path":"SimClr_train.py","file_url":"https://github.com/sprintml/memorizationssl/blob/HEAD/SimClr_train.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e12dc0e9953ba2ea"}},{"code_sha256_prefix":"a0a63b0e2f702d7c","entry":"train","repo":"sprintml/memorizationssl","repo_kind":"official","path":"SimClr_train.py","file_url":"https://github.com/sprintml/memorizationssl/blob/HEAD/SimClr_train.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a0a63b0e2f702d7c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}