{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/select-a-large-scale-benchmark-of-data","title":"SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image Classification","arxiv_id":"2410.05057","date":"2024-10-07","proceeding":null,"authors":["Benjamin Feuer","Jiawei Xu","Niv Cohen","Patrick Yubeaton","Govind Mittal","Chinmay Hegde"],"abstract":"Data curation is the problem of how to collect and organize samples into a dataset that supports efficient learning. Despite the centrality of the task, little work has been devoted towards a large-scale, systematic comparison of various curation methods. In this work, we take steps towards a formal evaluation of data curation strategies and introduce SELECT, the first large-scale benchmark of curation strategies for image classification. In order to generate baseline methods for the SELECT benchmark, we create a new dataset, ImageNet++, which constitutes the largest superset of ImageNet-1K to date. Our dataset extends ImageNet with 5 new training-data shifts, each approximately the size of ImageNet-1K itself, and each assembled using a distinct curation strategy. We evaluate our data curation baselines in two ways: (i) using each training-data shift to train identical image classification models from scratch (ii) using the data itself to fit a pretrained self-supervised representation. Our findings show interesting trends, particularly pertaining to recent methods for data curation such as synthetic data generation and lookup based on CLIP embeddings. We show that although these strategies are highly competitive for certain tasks, the curation strategy used to assemble the original ImageNet-1K dataset remains the gold standard. We anticipate that our benchmark can illuminate the path for new methods to further reduce the gap. We release our checkpoints, code, documentation, and a link to our dataset at https://github.com/jimmyxu123/SELECT.","url_abs":"https://arxiv.org/abs/2410.05057v1","url_pdf":"https://arxiv.org/pdf/2410.05057v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"select-a-large-scale-benchmark-of-data","repo_url":"https://github.com/jimmyxu123/select","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"synthetic-data-generation","task_name":"Synthetic Data Generation"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.05057","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.05057"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/jimmyxu123/SELECT","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jimmyxu123/select","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":10,"unverified":3},"by_repo_kind":{"official":{"samples":13,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"424fc184adb4a84c","entry":"dedup_features","repo":"jimmyxu123/SELECT","repo_kind":"official","path":"ssl/utils.py","file_url":"https://github.com/jimmyxu123/SELECT/blob/HEAD/ssl/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"424fc184adb4a84c"}},{"code_sha256_prefix":"1d74bc67f576b83a","entry":"get_class_labels","repo":"jimmyxu123/SELECT","repo_kind":"official","path":"ssl/reader_hfds.py","file_url":"https://github.com/jimmyxu123/SELECT/blob/HEAD/ssl/reader_hfds.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d74bc67f576b83a"}},{"code_sha256_prefix":"44aa6596c2101dce","entry":"get_mean_and_std","repo":"jimmyxu123/SELECT","repo_kind":"official","path":"vtab/getDataLoader.py","file_url":"https://github.com/jimmyxu123/SELECT/blob/HEAD/vtab/getDataLoader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44aa6596c2101dce"}},{"code_sha256_prefix":"2161fec9a47fe2fb","entry":"load_class_map","repo":"jimmyxu123/SELECT","repo_kind":"official","path":"ssl/reader_hfds.py","file_url":"https://github.com/jimmyxu123/SELECT/blob/HEAD/ssl/reader_hfds.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2161fec9a47fe2fb"}},{"code_sha256_prefix":"0a84b912bb131119","entry":"log_and_continue","repo":"jimmyxu123/SELECT","repo_kind":"official","path":"analytical/quality-metrics.py","file_url":"https://github.com/jimmyxu123/SELECT/blob/HEAD/analytical/quality-metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a84b912bb131119"}},{"code_sha256_prefix":"e313a5d6604a698a","entry":"regularize_label_count","repo":"jimmyxu123/SELECT","repo_kind":"official","path":"ssl/utils.py","file_url":"https://github.com/jimmyxu123/SELECT/blob/HEAD/ssl/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e313a5d6604a698a"}},{"code_sha256_prefix":"a1db47a39780034c","entry":"resample_features","repo":"jimmyxu123/SELECT","repo_kind":"official","path":"ssl/utils.py","file_url":"https://github.com/jimmyxu123/SELECT/blob/HEAD/ssl/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1db47a39780034c"}},{"code_sha256_prefix":"1b5c2979cb69dcb5","entry":"test_loop","repo":"jimmyxu123/SELECT","repo_kind":"official","path":"vtab/vtab_finetune.py","file_url":"https://github.com/jimmyxu123/SELECT/blob/HEAD/vtab/vtab_finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b5c2979cb69dcb5"}},{"code_sha256_prefix":"1c9147e50febcff4","entry":"to_device","repo":"jimmyxu123/SELECT","repo_kind":"official","path":"vtab/getDataLoader.py","file_url":"https://github.com/jimmyxu123/SELECT/blob/HEAD/vtab/getDataLoader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c9147e50febcff4"}},{"code_sha256_prefix":"7b10e462ddbf8e58","entry":"train_loop","repo":"jimmyxu123/SELECT","repo_kind":"official","path":"vtab/vtab_finetune.py","file_url":"https://github.com/jimmyxu123/SELECT/blob/HEAD/vtab/vtab_finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b10e462ddbf8e58"}},{"code_sha256_prefix":"c7c1bff321bc2884","entry":"easyrobust_eval","repo":"jimmyxu123/SELECT","repo_kind":"official","path":"base-ood/base_ood_eval.py","file_url":"https://github.com/jimmyxu123/SELECT/blob/HEAD/base-ood/base_ood_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7c1bff321bc2884"}},{"code_sha256_prefix":"e0126d3f3d3d4c42","entry":"get_data_loader","repo":"jimmyxu123/SELECT","repo_kind":"official","path":"vtab/getDataLoader.py","file_url":"https://github.com/jimmyxu123/SELECT/blob/HEAD/vtab/getDataLoader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e0126d3f3d3d4c42"}},{"code_sha256_prefix":"6ecb6791e6a9ebfb","entry":"knn_classifier","repo":"jimmyxu123/SELECT","repo_kind":"official","path":"ssl/eval_knn.py","file_url":"https://github.com/jimmyxu123/SELECT/blob/HEAD/ssl/eval_knn.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ecb6791e6a9ebfb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}