{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/variance-alignment-score-a-simple-but-tough","title":"Variance Alignment Score: A Simple But Tough-to-Beat Data Selection Method for Multimodal Contrastive Learning","arxiv_id":"2402.02055","date":"2024-02-03","proceeding":null,"authors":["Yiping Wang","Yifang Chen","Wendan Yan","Kevin Jamieson","Simon Shaolei Du"],"abstract":"In recent years, data selection has emerged as a core issue for large-scale visual-language model pretraining, especially on noisy web-curated datasets. One widely adopted strategy assigns quality scores such as CLIP similarity for each sample and retains the data pairs with the highest scores. However, these approaches are agnostic of data distribution and always fail to select the most informative samples. To solve this problem, we propose a simple yet theoretically principled metric named Variance Alignment Score (VAS), which has the form $\\langle \\Sigma_{\\text{test}}, \\Sigma_i\\rangle$. Here, $\\Sigma_{\\text{test}}$ represents the target (cross-)covariance matrix we aim to align, potentially based on prior knowledge, while $\\Sigma_i$ denotes the tensor product of single or multi-modal representations for the $i$-th sample. We further design a new data selection method that maximizes the total VAS. We provide theoretical analysis in a simplified setting to demonstrate the theoretical advantage of VAS over random or other existing data selection. Experimentally, applying VAS and CLIP scores together can outperform baselines by a margin of $1.3\\%$ average on 38 evaluation sets for noisy dataset DataComp and $2.5\\%$ on VTAB for high-quality dataset CC12M. Additionally, our ablation study also shows visual features are better than text for calculating VAS, and the related classical experimental design methods may fail under this context.","url_abs":"https://arxiv.org/abs/2402.02055v1","url_pdf":"https://arxiv.org/pdf/2402.02055v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"variance-alignment-score-a-simple-but-tough","repo_url":"https://github.com/ypwang61/VAS","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"variance-alignment-score-a-simple-but-tough","repo_url":"https://github.com/ypwang61/negcliploss_normsim","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"experimental-design","task_name":"Experimental Design"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.02055","atlas_url":"https://app.syntology.ai/?focus=2402.02055","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.02055"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ypwang61/negcliploss_normsim","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ypwang61/VAS","reach":{"status":"ok"}}],"summary":{"ran_violates":3},"by_repo_kind":{"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"f889070e0cb98b2f","entry":"get_ranks","repo":"ypwang61/negcliploss_normsim","repo_kind":"listed","path":"baselines/vas2.py","file_url":"https://github.com/ypwang61/negcliploss_normsim/blob/HEAD/baselines/vas2.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f889070e0cb98b2f"}},{"code_sha256_prefix":"2335a1161936c9fa","entry":"get_ranks_ratios","repo":"ypwang61/negcliploss_normsim","repo_kind":"listed","path":"baselines/vas2.py","file_url":"https://github.com/ypwang61/negcliploss_normsim/blob/HEAD/baselines/vas2.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2335a1161936c9fa"}},{"code_sha256_prefix":"8ec35d035919e5bc","entry":"get_small_ranks","repo":"ypwang61/negcliploss_normsim","repo_kind":"listed","path":"baselines/vas2.py","file_url":"https://github.com/ypwang61/negcliploss_normsim/blob/HEAD/baselines/vas2.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8ec35d035919e5bc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}