{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/when-do-we-not-need-larger-vision-models","title":"When Do We Not Need Larger Vision Models?","arxiv_id":"2403.13043","date":"2024-03-19","proceeding":null,"authors":["Baifeng Shi","Ziyang Wu","Maolin Mao","Xin Wang","Trevor Darrell"],"abstract":"Scaling up the size of vision models has been the de facto standard to obtain more powerful visual representations. In this work, we discuss the point beyond which larger vision models are not necessary. First, we demonstrate the power of Scaling on Scales (S$^2$), whereby a pre-trained and frozen smaller vision model (e.g., ViT-B or ViT-L), run over multiple image scales, can outperform larger models (e.g., ViT-H or ViT-G) on classification, segmentation, depth estimation, Multimodal LLM (MLLM) benchmarks, and robotic manipulation. Notably, S$^2$ achieves state-of-the-art performance in detailed understanding of MLLM on the V* benchmark, surpassing models such as GPT-4V. We examine the conditions under which S$^2$ is a preferred scaling approach compared to scaling on model size. While larger models have the advantage of better generalization on hard examples, we show that features of larger vision models can be well approximated by those of multi-scale smaller models. This suggests most, if not all, of the representations learned by current large pre-trained models can also be obtained from multi-scale smaller models. Our results show that a multi-scale smaller model has comparable learning capacity to a larger model, and pre-training smaller models with S$^2$ can match or even exceed the advantage of larger models. We release a Python package that can apply S$^2$ on any vision model with one line of code: https://github.com/bfshi/scaling_on_scales.","url_abs":"https://arxiv.org/abs/2403.13043v2","url_pdf":"https://arxiv.org/pdf/2403.13043v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"when-do-we-not-need-larger-vision-models","repo_url":"https://github.com/bfshi/scaling_on_scales","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"when-do-we-not-need-larger-vision-models","repo_url":"https://github.com/cubiq/comfyui_ipadapter_plus","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[{"task_slug":"depth-estimation","task_name":"Depth Estimation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2403.13043","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.13043"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cubiq/comfyui_ipadapter_plus","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/bfshi/scaling_on_scales","reach":null}],"summary":{"ran_fixture":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"05dd0f772495786e","entry":"batched_forward","repo":"bfshi/scaling_on_scales","repo_kind":"official","path":"s2wrapper/utils.py","file_url":"https://github.com/bfshi/scaling_on_scales/blob/HEAD/s2wrapper/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"05dd0f772495786e"}},{"code_sha256_prefix":"1f72e5980923f93b","entry":"merge_chessboard","repo":"bfshi/scaling_on_scales","repo_kind":"official","path":"s2wrapper/utils.py","file_url":"https://github.com/bfshi/scaling_on_scales/blob/HEAD/s2wrapper/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1f72e5980923f93b"}},{"code_sha256_prefix":"11473e6d969f3211","entry":"split_chessboard","repo":"bfshi/scaling_on_scales","repo_kind":"official","path":"s2wrapper/utils.py","file_url":"https://github.com/bfshi/scaling_on_scales/blob/HEAD/s2wrapper/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11473e6d969f3211"}},{"code_sha256_prefix":"33d75f66e6e135e7","entry":"forward","repo":"bfshi/scaling_on_scales","repo_kind":"official","path":"s2wrapper/core.py","file_url":"https://github.com/bfshi/scaling_on_scales/blob/HEAD/s2wrapper/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33d75f66e6e135e7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}