{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reproducible-scaling-laws-for-contrastive","title":"Reproducible scaling laws for contrastive language-image learning","arxiv_id":"2212.07143","date":"2022-12-14","proceeding":"CVPR 2023 1","authors":["Mehdi Cherti","Romain Beaumont","Ross Wightman","Mitchell Wortsman","Gabriel Ilharco","Cade Gordon","Christoph Schuhmann","Ludwig Schmidt","Jenia Jitsev"],"abstract":"Scaling up neural networks has led to remarkable performance across a wide range of tasks. Moreover, performance often follows reliable scaling laws as a function of training set size, model size, and compute, which offers valuable guidance as large-scale experiments are becoming increasingly expensive. However, previous work on scaling laws has primarily used private data \\& models or focused on uni-modal language or vision learning. To address these limitations, we investigate scaling laws for contrastive language-image pre-training (CLIP) with the public LAION dataset and the open-source OpenCLIP repository. Our large-scale experiments involve models trained on up to two billion image-text pairs and identify power law scaling for multiple downstream tasks including zero-shot classification, retrieval, linear probing, and end-to-end fine-tuning. We find that the training distribution plays a key role in scaling laws as the OpenAI and OpenCLIP models exhibit different scaling behavior despite identical model architectures and similar training recipes. We open-source our evaluation workflow and all models, including the largest public CLIP models, to ensure reproducibility and make scaling laws research more accessible. Source code and instructions to reproduce this study will be available at https://github.com/LAION-AI/scaling-laws-openclip","url_abs":"https://arxiv.org/abs/2212.07143v2","url_pdf":"https://arxiv.org/pdf/2212.07143v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reproducible-scaling-laws-for-contrastive","repo_url":"https://github.com/laion-ai/scaling-laws-openclip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"reproducible-scaling-laws-for-contrastive","repo_url":"https://github.com/eify/open_clip","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"reproducible-scaling-laws-for-contrastive","repo_url":"https://github.com/mlfoundations/open_clip","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"reproducible-scaling-laws-for-contrastive","repo_url":"https://github.com/nahidalam/open_clip","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"reproducible-scaling-laws-for-contrastive","repo_url":"https://github.com/shkarupa-alex/tfclip","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"open-vocabulary-attribute-detection","task_name":"Open Vocabulary Attribute Detection"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"zero-shot-cross-modal-retrieval","task_name":"Zero-Shot Cross-Modal Retrieval"},{"task_slug":"zero-shot-image-classification","task_name":"Zero-Shot Image Classification"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":null,"task_name":"zero-shot-classification"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"OpenCLIP ViT-H/14","rank_in_archive_order":42,"of":1060,"metrics":{"Top 1 Accuracy":"88.5%"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-attribute-detection-on-ovad-1","task":"Open Vocabulary Attribute Detection","dataset":"OVAD-Box benchmark","model":"Open CLIP ViT-B32","rank_in_archive_order":6,"of":7,"metrics":{"mean average precision":"17.0"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-cross-modal-retrieval-on-flickr30k","task":"Zero-Shot Cross-Modal Retrieval","dataset":"Flickr30k","model":"OpenCLIP VIT-H/14","rank_in_archive_order":21,"of":22,"metrics":{"Image-to-text R@1":"-","Image-to-text R@10":"-","Image-to-text R@5":"99.3","Text-to-image R@1":"-","Text-to-image R@10":"-","Text-to-image R@5":"94.1"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-image-classification-on-country211","task":"Zero-Shot Image Classification","dataset":"Country211","model":"OpenClip H/14 (34B)(Laion2B)","rank_in_archive_order":1,"of":1,"metrics":{"Top-1 accuracy":"30.01"},"uses_additional_data":true}],"syntology":{"syntology_url":"https://syntology.ai/paper/2212.07143","atlas_url":"https://app.syntology.ai/?focus=2212.07143","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.07143"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mlfoundations/open_clip","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/laion-ai/scaling-laws-openclip","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eify/open_clip","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shkarupa-alex/tfclip","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nahidalam/open_clip","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/LAION-AI/scaling-laws-openclip","reach":{"status":"ok"}}],"summary":{"ran_violates":1,"ran_draft_wrong":2},"by_repo_kind":{},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"f19962ebb134b3d7","entry":"convert_to_custom_text_state_dict","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"f19962ebb134b3d7"}},{"code_sha256_prefix":"dcd422d66b0581d8","entry":"get_cast_dtype","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"dcd422d66b0581d8"}},{"code_sha256_prefix":"b476c8cfbf0f1f47","entry":"get_input_dtype","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"b476c8cfbf0f1f47"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}