{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/self-supervised-learning-with-kernel","title":"Self-Supervised Learning with Kernel Dependence Maximization","arxiv_id":"2106.08320","date":"2021-06-15","proceeding":"NeurIPS 2021 12","authors":["Yazhe Li","Roman Pogodin","Danica J. Sutherland","Arthur Gretton"],"abstract":"We approach self-supervised learning of image representations from a statistical dependence perspective, proposing Self-Supervised Learning with the Hilbert-Schmidt Independence Criterion (SSL-HSIC). SSL-HSIC maximizes dependence between representations of transformations of an image and the image identity, while minimizing the kernelized variance of those representations. This framework yields a new understanding of InfoNCE, a variational lower bound on the mutual information (MI) between different transformations. While the MI itself is known to have pathologies which can result in learning meaningless representations, its bound is much better behaved: we show that it implicitly approximates SSL-HSIC (with a slightly different regularizer). Our approach also gives us insight into BYOL, a negative-free SSL method, since SSL-HSIC similarly learns local neighborhoods of samples. SSL-HSIC allows us to directly optimize statistical dependence in time linear in the batch size, without restrictive data assumptions or indirect mutual information estimators. Trained with or without a target network, SSL-HSIC matches the current state-of-the-art for standard linear evaluation on ImageNet, semi-supervised learning and transfer to other classification and vision tasks such as semantic segmentation, depth estimation and object recognition. Code is available at https://github.com/deepmind/ssl_hsic .","url_abs":"https://arxiv.org/abs/2106.08320v2","url_pdf":"https://arxiv.org/pdf/2106.08320v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"self-supervised-learning-with-kernel","repo_url":"https://github.com/deepmind/ssl_hsic","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":"linear-evaluation","task_name":"Linear evaluation"},{"task_slug":"object-recognition","task_name":"Object Recognition"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"byol","method_name":"BYOL"},{"method_slug":"infonce","method_name":"InfoNCE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.08320","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.08320"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepmind/ssl_hsic","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0688ca3fb68dd609","entry":"all_gather","repo":"deepmind/ssl_hsic","repo_kind":"official","path":"ssl_hsic/helpers.py","file_url":"https://github.com/deepmind/ssl_hsic/blob/HEAD/ssl_hsic/helpers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0688ca3fb68dd609"}},{"code_sha256_prefix":"bf31cca50c4f8a4b","entry":"compute_prob","repo":"deepmind/ssl_hsic","repo_kind":"official","path":"ssl_hsic/kernels.py","file_url":"https://github.com/deepmind/ssl_hsic/blob/HEAD/ssl_hsic/kernels.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bf31cca50c4f8a4b"}},{"code_sha256_prefix":"26fe2e60e0128a1d","entry":"get_label_weights","repo":"deepmind/ssl_hsic","repo_kind":"official","path":"ssl_hsic/kernels.py","file_url":"https://github.com/deepmind/ssl_hsic/blob/HEAD/ssl_hsic/kernels.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"26fe2e60e0128a1d"}},{"code_sha256_prefix":"4820794f76936b41","entry":"pairwise_distance_square","repo":"deepmind/ssl_hsic","repo_kind":"official","path":"ssl_hsic/kernels.py","file_url":"https://github.com/deepmind/ssl_hsic/blob/HEAD/ssl_hsic/kernels.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4820794f76936b41"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}