{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/generalization-beyond-data-imbalance-a","title":"What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights","arxiv_id":"2405.21070","date":"2024-05-31","proceeding":null,"authors":["Xin Wen","Bingchen Zhao","Yilun Chen","Jiangmiao Pang","Xiaojuan Qi"],"abstract":"Severe data imbalance naturally exists among web-scale vision-language datasets. Despite this, we find CLIP pre-trained thereupon exhibits notable robustness to the data imbalance compared to supervised learning, and demonstrates significant effectiveness in learning generalizable representations. With an aim to investigate the reasons behind this finding, we conduct controlled experiments to study various underlying factors, and reveal that CLIP's pretext task forms a dynamic classification problem wherein only a subset of classes is present in training. This isolates the bias from dominant classes and implicitly balances the learning signal. Furthermore, the robustness and discriminability of CLIP improve with more descriptive language supervision, larger data scale, and broader open-world concepts, which are inaccessible to supervised learning. Our study not only uncovers the mechanisms behind CLIP's generalizability beyond data imbalance but also provides transferable insights for the research community. The findings are validated in both supervised and self-supervised learning, enabling models trained on imbalanced data to achieve CLIP-level performance on diverse recognition tasks. Code and data are available at: https://github.com/CVMI-Lab/clip-beyond-tail.","url_abs":"https://arxiv.org/abs/2405.21070v3","url_pdf":"https://arxiv.org/pdf/2405.21070v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"generalization-beyond-data-imbalance-a","repo_url":"https://github.com/cvmi-lab/clip-beyond-tail","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"descriptive","task_name":"Descriptive"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2405.21070","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.21070"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/CVMI-Lab/clip-beyond-tail","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cvmi-lab/clip-beyond-tail","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_draft_wrong":2,"unverified":7},"by_repo_kind":{"official":{"samples":12,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8d3bee52aa79681c","entry":"calc_frequency","repo":"cvmi-lab/clip-beyond-tail","repo_kind":"official","path":"concept_freq_utils/calc_word_frequency.py","file_url":"https://github.com/cvmi-lab/clip-beyond-tail/blob/HEAD/concept_freq_utils/calc_word_frequency.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d3bee52aa79681c"}},{"code_sha256_prefix":"c744c8b07a785dcd","entry":"join_caption","repo":"cvmi-lab/clip-beyond-tail","repo_kind":"official","path":"data_utils/create_incaps_caption_variants.py","file_url":"https://github.com/cvmi-lab/clip-beyond-tail/blob/HEAD/data_utils/create_incaps_caption_variants.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c744c8b07a785dcd"}},{"code_sha256_prefix":"be4e00be50540042","entry":"log_and_continue","repo":"cvmi-lab/clip-beyond-tail","repo_kind":"official","path":"data_utils/arrange_laionet.py","file_url":"https://github.com/cvmi-lab/clip-beyond-tail/blob/HEAD/data_utils/arrange_laionet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be4e00be50540042"}},{"code_sha256_prefix":"b1022da5ff456771","entry":"parse_cmd","repo":"CVMI-Lab/clip-beyond-tail","repo_kind":"official","path":"exps_clip/run_pretrained_openclip.py","file_url":"https://github.com/CVMI-Lab/clip-beyond-tail/blob/HEAD/exps_clip/run_pretrained_openclip.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1022da5ff456771"}},{"code_sha256_prefix":"dcc0f8a3d856ad1f","entry":"sample_class_inds","repo":"CVMI-Lab/clip-beyond-tail","repo_kind":"official","path":"exps_sup/train_sup.py","file_url":"https://github.com/CVMI-Lab/clip-beyond-tail/blob/HEAD/exps_sup/train_sup.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dcc0f8a3d856ad1f"}},{"code_sha256_prefix":"083dd12611928e3f","entry":"calc_frequency","repo":"CVMI-Lab/clip-beyond-tail","repo_kind":"official","path":"concept_freq_utils/calc_class_frequency.py","file_url":"https://github.com/CVMI-Lab/clip-beyond-tail/blob/HEAD/concept_freq_utils/calc_class_frequency.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"083dd12611928e3f"}},{"code_sha256_prefix":"1f8b5c15c1b07e8c","entry":"calc_frequency","repo":"cvmi-lab/clip-beyond-tail","repo_kind":"official","path":"concept_freq_utils/calc_class_frequency_metaclip.py","file_url":"https://github.com/cvmi-lab/clip-beyond-tail/blob/HEAD/concept_freq_utils/calc_class_frequency_metaclip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1f8b5c15c1b07e8c"}},{"code_sha256_prefix":"7327f0f3714b44d0","entry":"get_synset","repo":"cvmi-lab/clip-beyond-tail","repo_kind":"official","path":"data_utils/create_incaps_caption_variants.py","file_url":"https://github.com/cvmi-lab/clip-beyond-tail/blob/HEAD/data_utils/create_incaps_caption_variants.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7327f0f3714b44d0"}},{"code_sha256_prefix":"478bab57fe97f4fd","entry":"preprocess_template","repo":"cvmi-lab/clip-beyond-tail","repo_kind":"official","path":"concept_freq_utils/calc_class_frequency.py","file_url":"https://github.com/cvmi-lab/clip-beyond-tail/blob/HEAD/concept_freq_utils/calc_class_frequency.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"478bab57fe97f4fd"}},{"code_sha256_prefix":"9928091c045bd498","entry":"preprocess_template","repo":"cvmi-lab/clip-beyond-tail","repo_kind":"official","path":"concept_freq_utils/calc_word_frequency.py","file_url":"https://github.com/cvmi-lab/clip-beyond-tail/blob/HEAD/concept_freq_utils/calc_word_frequency.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9928091c045bd498"}},{"code_sha256_prefix":"18545f8c98643da8","entry":"preprocess_text","repo":"cvmi-lab/clip-beyond-tail","repo_kind":"official","path":"concept_freq_utils/calc_class_frequency.py","file_url":"https://github.com/cvmi-lab/clip-beyond-tail/blob/HEAD/concept_freq_utils/calc_class_frequency.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"18545f8c98643da8"}},{"code_sha256_prefix":"bd6c9c789d1ef16d","entry":"rn50","repo":"cvmi-lab/clip-beyond-tail","repo_kind":"official","path":"exps_sup/model.py","file_url":"https://github.com/cvmi-lab/clip-beyond-tail/blob/HEAD/exps_sup/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd6c9c789d1ef16d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}