{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/does-language-help-generalization-in-vision","title":"Does language help generalization in vision models?","arxiv_id":"2104.08313","date":"2021-04-16","proceeding":"CoNLL (EMNLP) 2021 11","authors":["Benjamin Devillers","Bhavin Choksi","Romain Bielawski","Rufin VanRullen"],"abstract":"Vision models trained on multimodal datasets can benefit from the wide availability of large image-caption datasets. A recent model (CLIP) was found to generalize well in zero-shot and transfer learning settings. This could imply that linguistic or \"semantic grounding\" confers additional generalization abilities to the visual feature space. Here, we systematically evaluate various multimodal architectures and vision-only models in terms of unsupervised clustering, few-shot learning, transfer learning and adversarial robustness. In each setting, multimodal training produced no additional generalization capability compared to standard supervised visual training. We conclude that work is still required for semantic grounding to help improve vision models.","url_abs":"https://arxiv.org/abs/2104.08313v3","url_pdf":"https://arxiv.org/pdf/2104.08313v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"does-language-help-generalization-in-vision","repo_url":"https://github.com/bdvllrs/generalization-vision","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"},{"task_slug":"clustering","task_name":"Clustering"},{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2104.08313","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.08313"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bdvllrs/generalization-vision","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":1,"ran":1,"ran_honours":2},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"56f3e936358bba07","entry":"cosine_metric","repo":"bdvllrs/generalization-vision","repo_kind":"official","path":"tasks/unsupervised_clustering.py","file_url":"https://github.com/bdvllrs/generalization-vision/blob/HEAD/tasks/unsupervised_clustering.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"56f3e936358bba07"}},{"code_sha256_prefix":"f4b8cc02d399b6be","entry":"permute_labels","repo":"bdvllrs/generalization-vision","repo_kind":"official","path":"tasks/unsupervised_clustering.py","file_url":"https://github.com/bdvllrs/generalization-vision/blob/HEAD/tasks/unsupervised_clustering.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f4b8cc02d399b6be"}},{"code_sha256_prefix":"a59d584793b07994","entry":"train","repo":"bdvllrs/generalization-vision","repo_kind":"official","path":"tasks/transfer_learning.py","file_url":"https://github.com/bdvllrs/generalization-vision/blob/HEAD/tasks/transfer_learning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a59d584793b07994"}},{"code_sha256_prefix":"f67d457854b1f0dd","entry":"train_one_epoch","repo":"bdvllrs/generalization-vision","repo_kind":"official","path":"tasks/transfer_learning.py","file_url":"https://github.com/bdvllrs/generalization-vision/blob/HEAD/tasks/transfer_learning.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f67d457854b1f0dd"}},{"code_sha256_prefix":"4f69a77f04e33665","entry":"val","repo":"bdvllrs/generalization-vision","repo_kind":"official","path":"tasks/transfer_learning.py","file_url":"https://github.com/bdvllrs/generalization-vision/blob/HEAD/tasks/transfer_learning.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f69a77f04e33665"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}