{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/automatic-data-curation-for-self-supervised","title":"Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach","arxiv_id":"2405.15613","date":"2024-05-24","proceeding":null,"authors":["Huy V. Vo","Vasil Khalidov","Timothée Darcet","Théo Moutakanni","Nikita Smetanin","Marc Szafraniec","Hugo Touvron","Camille Couprie","Maxime Oquab","Armand Joulin","Hervé Jégou","Patrick Labatut","Piotr Bojanowski"],"abstract":"Self-supervised features are the cornerstone of modern machine learning systems. They are typically pre-trained on data collections whose construction and curation typically require extensive human effort. This manual process has some limitations similar to those encountered in supervised learning, e.g., the crowd-sourced selection of data is costly and time-consuming, preventing scaling the dataset size. In this work, we consider the problem of automatic curation of high-quality datasets for self-supervised pre-training. We posit that such datasets should be large, diverse and balanced, and propose a clustering-based approach for building ones satisfying all these criteria. Our method involves successive and hierarchical applications of $k$-means on a large and diverse data repository to obtain clusters that distribute uniformly among data concepts, followed by a hierarchical, balanced sampling step from these clusters. Extensive experiments on three different data domains including web-based images, satellite images and text show that features trained on our automatically curated datasets outperform those trained on uncurated data while being on par or better than ones trained on manually curated data. Code is available at https://github.com/facebookresearch/ssl-data-curation.","url_abs":"https://arxiv.org/abs/2405.15613v2","url_pdf":"https://arxiv.org/pdf/2405.15613v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"automatic-data-curation-for-self-supervised","repo_url":"https://github.com/facebookresearch/ssl-data-curation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"clustering","task_name":"Clustering"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.15613","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.15613"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/ssl-data-curation","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":15,"unverified":4},"by_repo_kind":{"official":{"samples":19,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":19,"samples":[{"code_sha256_prefix":"5f9322e34f0f5a5a","entry":"closest_to_centroid_selection","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"src/hierarchical_sampling.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/src/hierarchical_sampling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5f9322e34f0f5a5a"}},{"code_sha256_prefix":"f0ba8fa85b2c379a","entry":"compute_distance","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"src/kmeans_gpu.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/src/kmeans_gpu.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f0ba8fa85b2c379a"}},{"code_sha256_prefix":"2c6f00b7dd49dbfd","entry":"create_clusters_from_cluster_assignment","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2c6f00b7dd49dbfd"}},{"code_sha256_prefix":"28a739ca57e0edb4","entry":"find_all_checkpoints","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"28a739ca57e0edb4"}},{"code_sha256_prefix":"5a8571a0cd12ae5b","entry":"find_subcluster_target_size","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"src/hierarchical_sampling.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/src/hierarchical_sampling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5a8571a0cd12ae5b"}},{"code_sha256_prefix":"3e61ccbd2cdd0e61","entry":"get_last_valid_checkpoint","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3e61ccbd2cdd0e61"}},{"code_sha256_prefix":"904db215a04accee","entry":"get_part_indices","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"src/distributed_kmeans_gpu.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/src/distributed_kmeans_gpu.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"904db215a04accee"}},{"code_sha256_prefix":"bc6694c0ea0d2cb1","entry":"get_part_len","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"src/distributed_kmeans_gpu.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/src/distributed_kmeans_gpu.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"bc6694c0ea0d2cb1"}},{"code_sha256_prefix":"4c8210bf31a58081","entry":"kmeans_plusplus","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"src/kmeans_gpu.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/src/kmeans_gpu.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4c8210bf31a58081"}},{"code_sha256_prefix":"060b6e8429938eef","entry":"l2_squared_power","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"vis/generalized_kmeans_1d.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/vis/generalized_kmeans_1d.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"060b6e8429938eef"}},{"code_sha256_prefix":"77f6fab0a9920b2d","entry":"l2_squared_power_der","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"vis/generalized_kmeans_1d.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/vis/generalized_kmeans_1d.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"77f6fab0a9920b2d"}},{"code_sha256_prefix":"1d6eb8e587e8f8e8","entry":"l2_squared_power_der2","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"vis/generalized_kmeans_1d.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/vis/generalized_kmeans_1d.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1d6eb8e587e8f8e8"}},{"code_sha256_prefix":"b5d1408c7aaf8dac","entry":"load_clusters_from_file","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"src/clusters.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/src/clusters.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b5d1408c7aaf8dac"}},{"code_sha256_prefix":"ce165668d59de980","entry":"matmul_transpose","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"src/kmeans_gpu.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/src/kmeans_gpu.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ce165668d59de980"}},{"code_sha256_prefix":"b4b479fcfe0ff407","entry":"random_selection","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"src/hierarchical_sampling.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/src/hierarchical_sampling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b4b479fcfe0ff407"}},{"code_sha256_prefix":"b4ac6b51806c8b28","entry":"gather_tensor","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"src/dist_comm.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/src/dist_comm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b4ac6b51806c8b28"}},{"code_sha256_prefix":"9392b81c15f05e2a","entry":"hierarchical_kmeans","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"src/hierarchical_kmeans_gpu.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/src/hierarchical_kmeans_gpu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9392b81c15f05e2a"}},{"code_sha256_prefix":"410b7cf8ca0cdf31","entry":"hierarchical_kmeans_with_resampling","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"src/hierarchical_kmeans_gpu.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/src/hierarchical_kmeans_gpu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"410b7cf8ca0cdf31"}},{"code_sha256_prefix":"96c349a4812a0b80","entry":"load_data_to_worker","repo":"facebookresearch/ssl-data-curation","repo_kind":"official","path":"src/distributed_kmeans_gpu.py","file_url":"https://github.com/facebookresearch/ssl-data-curation/blob/HEAD/src/distributed_kmeans_gpu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"96c349a4812a0b80"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}