{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cliploss-and-norm-based-data-selection","title":"CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning","arxiv_id":"2405.19547","date":"2024-05-29","proceeding":null,"authors":["Yiping Wang","Yifang Chen","Wendan Yan","Alex Fang","Wenjing Zhou","Kevin Jamieson","Simon Shaolei Du"],"abstract":"Data selection has emerged as a core issue for large-scale visual-language model pretaining (e.g., CLIP), particularly with noisy web-curated datasets. Three main data selection approaches are: (1) leveraging external non-CLIP models to aid data selection, (2) training new CLIP-style embedding models that are more effective at selecting high-quality data than the original OpenAI CLIP model, and (3) designing better metrics or strategies universally applicable to any CLIP embedding without requiring specific model properties (e.g., CLIPScore is one popular metric). While the first two approaches have been extensively studied, the third remains under-explored. In this paper, we advance the third approach by proposing two new methods. Firstly, instead of classical CLIP scores that only consider the alignment between two modalities from a single sample, we introduce surrogate-CLIPLoss (s-CLIPLoss), a CLIP loss-inspired method that adds the alignment between one sample and its contrastive pairs as an extra normalization term for better quality measurement. Secondly, when downstream tasks are known, we propose a new norm-based metric, NormSim, to measure the similarity between pretraining data and target data. We test our methods on the data selection benchmark, DataComp~\\cite{gadre2023datacomp}. Compared to the best baseline using only OpenAI's CLIP-L/14, our methods achieve a 5.3\\% improvement on ImageNet-1k and a 2.8\\% improvement on 38 downstream evaluation tasks. Moreover, both s-CLIPLoss and NormSim are compatible with existing techniques. By combining our methods with the current best methods DFN and HYPE, we can boost average performance on downstream tasks by 0.9\\%, achieving a new state-of-the-art on the DataComp-medium benchmark.","url_abs":"https://arxiv.org/abs/2405.19547v2","url_pdf":"https://arxiv.org/pdf/2405.19547v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cliploss-and-norm-based-data-selection","repo_url":"https://github.com/ypwang61/negcliploss_normsim","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"cliploss-and-norm-based-data-selection","repo_url":"https://github.com/ypwang61/VAS","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.19547","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.19547"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ypwang61/negcliploss_normsim","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ypwang61/VAS","reach":{"status":"ok"}}],"summary":{"ran":9,"ran_violates":3,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":16,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":19,"samples":[{"code_sha256_prefix":"00dd429e9775e127","entry":"available_scales","repo":"ypwang61/VAS","repo_kind":"official","path":"scale_configs.py","file_url":"https://github.com/ypwang61/VAS/blob/HEAD/scale_configs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"00dd429e9775e127"}},{"code_sha256_prefix":"53deafe2a17c9bd5","entry":"filter_by_score","repo":"ypwang61/VAS","repo_kind":"official","path":"baselines/vas.py","file_url":"https://github.com/ypwang61/VAS/blob/HEAD/baselines/vas.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"53deafe2a17c9bd5"}},{"code_sha256_prefix":"6b052976ecde8c23","entry":"generate_p_from_clipscores","repo":"ypwang61/VAS","repo_kind":"official","path":"cal_clipscores.py","file_url":"https://github.com/ypwang61/VAS/blob/HEAD/cal_clipscores.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b052976ecde8c23"}},{"code_sha256_prefix":"83f216dbfcc79287","entry":"get_aggregate_scores","repo":"ypwang61/VAS","repo_kind":"official","path":"aggregate_scores.py","file_url":"https://github.com/ypwang61/VAS/blob/HEAD/aggregate_scores.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"83f216dbfcc79287"}},{"code_sha256_prefix":"61d5b2eae57f28be","entry":"get_fasttext_language","repo":"ypwang61/VAS","repo_kind":"official","path":"baselines/apply_filter.py","file_url":"https://github.com/ypwang61/VAS/blob/HEAD/baselines/apply_filter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61d5b2eae57f28be"}},{"code_sha256_prefix":"6cc97b780724c9b6","entry":"get_gcld3_language","repo":"ypwang61/VAS","repo_kind":"official","path":"baselines/apply_filter.py","file_url":"https://github.com/ypwang61/VAS/blob/HEAD/baselines/apply_filter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6cc97b780724c9b6"}},{"code_sha256_prefix":"f889070e0cb98b2f","entry":"get_ranks","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"f889070e0cb98b2f"}},{"code_sha256_prefix":"2335a1161936c9fa","entry":"get_ranks_ratios","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"2335a1161936c9fa"}},{"code_sha256_prefix":"673bbfe27cf768f1","entry":"get_scale_config","repo":"ypwang61/VAS","repo_kind":"official","path":"scale_configs.py","file_url":"https://github.com/ypwang61/VAS/blob/HEAD/scale_configs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"673bbfe27cf768f1"}},{"code_sha256_prefix":"8ec35d035919e5bc","entry":"get_small_ranks","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"8ec35d035919e5bc"}},{"code_sha256_prefix":"63f3d769047ceab6","entry":"get_vas_gpu","repo":"ypwang61/VAS","repo_kind":"official","path":"baselines/vas.py","file_url":"https://github.com/ypwang61/VAS/blob/HEAD/baselines/vas.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"63f3d769047ceab6"}},{"code_sha256_prefix":"cfd8b1cf3644b321","entry":"random_sample","repo":"ypwang61/VAS","repo_kind":"official","path":"cal_clipscores.py","file_url":"https://github.com/ypwang61/VAS/blob/HEAD/cal_clipscores.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cfd8b1cf3644b321"}},{"code_sha256_prefix":"28962743449aaadd","entry":"setup_process_logging","repo":"ypwang61/VAS","repo_kind":"official","path":"resharder.py","file_url":"https://github.com/ypwang61/VAS/blob/HEAD/resharder.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"28962743449aaadd"}},{"code_sha256_prefix":"bb98f2d1cb3c5ff8","entry":"download","repo":"ypwang61/VAS","repo_kind":"official","path":"baselines/utils.py","file_url":"https://github.com/ypwang61/VAS/blob/HEAD/baselines/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bb98f2d1cb3c5ff8"}},{"code_sha256_prefix":"b4ca49c3c0d1f7d0","entry":"filter_by_given_uids","repo":"ypwang61/VAS","repo_kind":"official","path":"baselines/vas.py","file_url":"https://github.com/ypwang61/VAS/blob/HEAD/baselines/vas.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b4ca49c3c0d1f7d0"}},{"code_sha256_prefix":"6da3dc3e075307c6","entry":"submit_to_firebase","repo":"ypwang61/VAS","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/ypwang61/VAS/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6da3dc3e075307c6"}},{"code_sha256_prefix":"85c09d957a5f2bc8","entry":"submit_to_slack","repo":"ypwang61/VAS","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/ypwang61/VAS/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"85c09d957a5f2bc8"}},{"code_sha256_prefix":"24e11664919d0016","entry":"train_kmeans","repo":"ypwang61/VAS","repo_kind":"official","path":"baselines/image_based_clustering.py","file_url":"https://github.com/ypwang61/VAS/blob/HEAD/baselines/image_based_clustering.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"24e11664919d0016"}},{"code_sha256_prefix":"fca45920dddd4939","entry":"worker_threadpool","repo":"ypwang61/VAS","repo_kind":"official","path":"baselines/utils.py","file_url":"https://github.com/ypwang61/VAS/blob/HEAD/baselines/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fca45920dddd4939"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}