{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vitamin-designing-scalable-vision-models-in","title":"ViTamin: Designing Scalable Vision Models in the Vision-Language Era","arxiv_id":"2404.02132","date":"2024-04-02","proceeding":"CVPR 2024 1","authors":["Jieneng Chen","Qihang Yu","Xiaohui Shen","Alan Yuille","Liang-Chieh Chen"],"abstract":"Recent breakthroughs in vision-language models (VLMs) start a new page in the vision community. The VLMs provide stronger and more generalizable feature embeddings compared to those from ImageNet-pretrained models, thanks to the training on the large-scale Internet image-text pairs. However, despite the amazing achievement from the VLMs, vanilla Vision Transformers (ViTs) remain the default choice for the image encoder. Although pure transformer proves its effectiveness in the text encoding area, it remains questionable whether it is also the case for image encoding, especially considering that various types of networks are proposed on the ImageNet benchmark, which, unfortunately, are rarely studied in VLMs. Due to small data/model scale, the original conclusions of model design on ImageNet can be limited and biased. In this paper, we aim at building an evaluation protocol of vision models in the vision-language era under the contrastive language-image pretraining (CLIP) framework. We provide a comprehensive way to benchmark different vision models, covering their zero-shot performance and scalability in both model and training data sizes. To this end, we introduce ViTamin, a new vision models tailored for VLMs. ViTamin-L significantly outperforms ViT-L by 2.0% ImageNet zero-shot accuracy, when using the same publicly available DataComp-1B dataset and the same OpenCLIP training scheme. ViTamin-L presents promising results on 60 diverse benchmarks, including classification, retrieval, open-vocabulary detection and segmentation, and large multi-modal models. When further scaling up the model size, our ViTamin-XL with only 436M parameters attains 82.9% ImageNet zero-shot accuracy, surpassing 82.0% achieved by EVA-E that has ten times more parameters (4.4B).","url_abs":"https://arxiv.org/abs/2404.02132v2","url_pdf":"https://arxiv.org/pdf/2404.02132v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vitamin-designing-scalable-vision-models-in","repo_url":"https://github.com/beckschen/vitamin","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"vitamin-designing-scalable-vision-models-in","repo_url":"https://github.com/MindCode-4/code-5/tree/main/vit_msn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.02132","atlas_url":"https://app.syntology.ai/?focus=2404.02132","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.02132"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/beckschen/vitamin","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindCode-4/code-5/tree/main/vit_msn","reach":null}],"summary":{"ran_violates":1,"ran_draft_wrong":2,"ran":1,"unverified":8},"by_repo_kind":{"official":{"samples":10,"ran":4,"repositories":1},"listed":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f19962ebb134b3d7","entry":"convert_to_custom_text_state_dict","repo":"beckschen/vitamin","repo_kind":"official","path":"ViTamin/open_clip/model.py","file_url":"https://github.com/beckschen/vitamin/blob/HEAD/ViTamin/open_clip/model.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f19962ebb134b3d7"}},{"code_sha256_prefix":"dcd422d66b0581d8","entry":"get_cast_dtype","repo":"beckschen/vitamin","repo_kind":"official","path":"ViTamin/open_clip/model.py","file_url":"https://github.com/beckschen/vitamin/blob/HEAD/ViTamin/open_clip/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dcd422d66b0581d8"}},{"code_sha256_prefix":"b476c8cfbf0f1f47","entry":"get_input_dtype","repo":"beckschen/vitamin","repo_kind":"official","path":"ViTamin/open_clip/model.py","file_url":"https://github.com/beckschen/vitamin/blob/HEAD/ViTamin/open_clip/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b476c8cfbf0f1f47"}},{"code_sha256_prefix":"4b3a4d17e1df3140","entry":"load_state_dict","repo":"beckschen/vitamin","repo_kind":"official","path":"ViTamin/open_clip/factory.py","file_url":"https://github.com/beckschen/vitamin/blob/HEAD/ViTamin/open_clip/factory.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4b3a4d17e1df3140"}},{"code_sha256_prefix":"ddcbd45e940484ee","entry":"gather_features","repo":"beckschen/vitamin","repo_kind":"official","path":"ViTamin/open_clip/loss.py","file_url":"https://github.com/beckschen/vitamin/blob/HEAD/ViTamin/open_clip/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ddcbd45e940484ee"}},{"code_sha256_prefix":"5c7cf820e99443ff","entry":"get_model_config","repo":"beckschen/vitamin","repo_kind":"official","path":"ViTamin/open_clip/factory.py","file_url":"https://github.com/beckschen/vitamin/blob/HEAD/ViTamin/open_clip/factory.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c7cf820e99443ff"}},{"code_sha256_prefix":"c55b52840039f922","entry":"get_tokenizer","repo":"beckschen/vitamin","repo_kind":"official","path":"ViTamin/open_clip/factory.py","file_url":"https://github.com/beckschen/vitamin/blob/HEAD/ViTamin/open_clip/factory.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c55b52840039f922"}},{"code_sha256_prefix":"8811c5d06f3a2eaa","entry":"load_openai_model","repo":"beckschen/vitamin","repo_kind":"official","path":"ViTamin/open_clip/openai.py","file_url":"https://github.com/beckschen/vitamin/blob/HEAD/ViTamin/open_clip/openai.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8811c5d06f3a2eaa"}},{"code_sha256_prefix":"fb651d0a97fd4d3f","entry":"prepare_inputs_for_generation","repo":"beckschen/vitamin","repo_kind":"official","path":"ViTamin/open_clip/coca_model.py","file_url":"https://github.com/beckschen/vitamin/blob/HEAD/ViTamin/open_clip/coca_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb651d0a97fd4d3f"}},{"code_sha256_prefix":"2a377da4a76a2d44","entry":"register_pooler","repo":"beckschen/vitamin","repo_kind":"official","path":"ViTamin/open_clip/hf_model.py","file_url":"https://github.com/beckschen/vitamin/blob/HEAD/ViTamin/open_clip/hf_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2a377da4a76a2d44"}},{"code_sha256_prefix":"46cc4b5e03a8eb4c","entry":"window_partition","repo":"MindCode-4/code-5","repo_kind":"listed","path":"swin/modeling_swin.py","file_url":"https://github.com/MindCode-4/code-5/blob/HEAD/swin/modeling_swin.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"46cc4b5e03a8eb4c"}},{"code_sha256_prefix":"4feb0a65bcd6ff42","entry":"window_reverse","repo":"MindCode-4/code-5","repo_kind":"listed","path":"swin/modeling_swin.py","file_url":"https://github.com/MindCode-4/code-5/blob/HEAD/swin/modeling_swin.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4feb0a65bcd6ff42"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}