{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/whitening-for-self-supervised-representation","title":"Whitening for Self-Supervised Representation Learning","arxiv_id":"2007.06346","date":"2020-07-13","proceeding":null,"authors":["Aleksandr Ermolov","Aliaksandr Siarohin","Enver Sangineto","Nicu Sebe"],"abstract":"Most of the current self-supervised representation learning (SSL) methods are based on the contrastive loss and the instance-discrimination task, where augmented versions of the same image instance (\"positives\") are contrasted with instances extracted from other images (\"negatives\"). For the learning to be effective, many negatives should be compared with a positive pair, which is computationally demanding. In this paper, we propose a different direction and a new loss function for SSL, which is based on the whitening of the latent-space features. The whitening operation has a \"scattering\" effect on the batch samples, avoiding degenerate solutions where all the sample representations collapse to a single point. Our solution does not require asymmetric networks and it is conceptually simple. Moreover, since negatives are not needed, we can extract multiple positive pairs from the same image instance. The source code of the method and of all the experiments is available at: https://github.com/htdt/self-supervised.","url_abs":"https://arxiv.org/abs/2007.06346v5","url_pdf":"https://arxiv.org/pdf/2007.06346v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"whitening-for-self-supervised-representation","repo_url":"https://github.com/htdt/self-supervised","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"whitening-for-self-supervised-representation","repo_url":"https://github.com/dr4thmos/solo-learn-radio","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"whitening-for-self-supervised-representation","repo_url":"https://github.com/lavoiems/simplicial-embeddings","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"whitening-for-self-supervised-representation","repo_url":"https://github.com/ludysama/crp","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"whitening-for-self-supervised-representation","repo_url":"https://github.com/trungpx/solo-learn-1214","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"whitening-for-self-supervised-representation","repo_url":"https://github.com/vturrisi/solo-learn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"whitening-for-self-supervised-representation","repo_url":"https://github.com/winci-ai/cw-rgp","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2007.06346","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2007.06346"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dr4thmos/solo-learn-radio","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vturrisi/solo-learn","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ludysama/crp","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lavoiems/simplicial-embeddings","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/winci-ai/cw-rgp","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/htdt/self-supervised","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/trungpx/solo-learn-1214","reach":null}],"summary":{"ran":8,"ran_draft_wrong":1,"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":12,"ran":9,"repositories":4}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"2fd07073535ad61b","entry":"Whitening2d","repo":"htdt/self-supervised","repo_kind":"official","path":"methods/whitening.py","file_url":"https://github.com/htdt/self-supervised/blob/HEAD/methods/whitening.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2fd07073535ad61b"}},{"code_sha256_prefix":"8e1c3896cc666ecd","entry":"Whitening2d","repo":"trungpx/solo-learn-1214","repo_kind":"listed","path":"solo/utils/whitening.py","file_url":"https://github.com/trungpx/solo-learn-1214/blob/HEAD/solo/utils/whitening.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e1c3896cc666ecd"}},{"code_sha256_prefix":"5ec9dc11fb54ff22","entry":"Whitening2d","repo":"vturrisi/solo-learn","repo_kind":"listed","path":"solo/utils/whitening.py","file_url":"https://github.com/vturrisi/solo-learn/blob/HEAD/solo/utils/whitening.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ec9dc11fb54ff22"}},{"code_sha256_prefix":"132a949a804eb81e","entry":"barlow_loss_func","repo":"dr4thmos/solo-learn-radio","repo_kind":"listed","path":"solo/losses/barlow.py","file_url":"https://github.com/dr4thmos/solo-learn-radio/blob/HEAD/solo/losses/barlow.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"132a949a804eb81e"}},{"code_sha256_prefix":"273532e06cb3967e","entry":"byol_loss_func","repo":"dr4thmos/solo-learn-radio","repo_kind":"listed","path":"solo/losses/byol.py","file_url":"https://github.com/dr4thmos/solo-learn-radio/blob/HEAD/solo/losses/byol.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"273532e06cb3967e"}},{"code_sha256_prefix":"a07d9a7525702b80","entry":"mae_loss_func","repo":"dr4thmos/solo-learn-radio","repo_kind":"listed","path":"solo/losses/mae.py","file_url":"https://github.com/dr4thmos/solo-learn-radio/blob/HEAD/solo/losses/mae.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a07d9a7525702b80"}},{"code_sha256_prefix":"4dd7c3cff0150cb9","entry":"mocov2plus_loss_func","repo":"dr4thmos/solo-learn-radio","repo_kind":"listed","path":"solo/losses/mocov2plus.py","file_url":"https://github.com/dr4thmos/solo-learn-radio/blob/HEAD/solo/losses/mocov2plus.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4dd7c3cff0150cb9"}},{"code_sha256_prefix":"3e6a557bf1ad83d9","entry":"patchify","repo":"dr4thmos/solo-learn-radio","repo_kind":"listed","path":"solo/losses/mae.py","file_url":"https://github.com/dr4thmos/solo-learn-radio/blob/HEAD/solo/losses/mae.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e6a557bf1ad83d9"}},{"code_sha256_prefix":"779f0ded23fd8106","entry":"ressl_loss_func","repo":"dr4thmos/solo-learn-radio","repo_kind":"listed","path":"solo/losses/ressl.py","file_url":"https://github.com/dr4thmos/solo-learn-radio/blob/HEAD/solo/losses/ressl.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"779f0ded23fd8106"}},{"code_sha256_prefix":"6b191c39aa0c6668","entry":"swav_loss_func","repo":"dr4thmos/solo-learn-radio","repo_kind":"listed","path":"solo/losses/swav.py","file_url":"https://github.com/dr4thmos/solo-learn-radio/blob/HEAD/solo/losses/swav.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b191c39aa0c6668"}},{"code_sha256_prefix":"d09f66fe441f6036","entry":"Whitening2d","repo":"winci-ai/cw-rgp","repo_kind":"listed","path":"ImageNet/methods/whitening.py","file_url":"https://github.com/winci-ai/cw-rgp/blob/HEAD/ImageNet/methods/whitening.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d09f66fe441f6036"}},{"code_sha256_prefix":"ce432f2e9653c7a3","entry":"deepclusterv2_loss_func","repo":"dr4thmos/solo-learn-radio","repo_kind":"listed","path":"solo/losses/deepclusterv2.py","file_url":"https://github.com/dr4thmos/solo-learn-radio/blob/HEAD/solo/losses/deepclusterv2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce432f2e9653c7a3"}},{"code_sha256_prefix":"cd64814bdf3905fa","entry":"simsiam_loss_func","repo":"dr4thmos/solo-learn-radio","repo_kind":"listed","path":"solo/losses/simsiam.py","file_url":"https://github.com/dr4thmos/solo-learn-radio/blob/HEAD/solo/losses/simsiam.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd64814bdf3905fa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}