{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/i-can-t-believe-there-s-no-images-learning","title":"I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervision","arxiv_id":"2211.09778","date":"2022-11-17","proceeding":"ICCV 2023 1","authors":["Sophia Gu","Christopher Clark","Aniruddha Kembhavi"],"abstract":"Many high-level skills that are required for computer vision tasks, such as parsing questions, comparing and contrasting semantics, and writing descriptions, are also required in other domains such as natural language processing. In this paper, we ask whether it is possible to learn those skills from text data and then transfer them to vision tasks without ever training on visual training data. Key to our approach is exploiting the joint embedding space of contrastively trained vision and language encoders. In practice, there can be systematic differences between embedding spaces for different modalities in contrastive models, and we analyze how these differences affect our approach and study strategies to mitigate this concern. We produce models using only text training data on four representative tasks: image captioning, visual entailment, visual question answering and visual news captioning, and evaluate them on standard benchmarks using images. We find these models perform close to models trained on images, while surpassing prior work for captioning and visual entailment in this text-only setting by over 9 points, and outperforming all prior work on visual news by over 30 points. We also showcase a variety of stylistic image captioning models that are trained using no image data and no human-curated language data, but instead using readily-available text data from books, the web, or language models.","url_abs":"https://arxiv.org/abs/2211.09778v4","url_pdf":"https://arxiv.org/pdf/2211.09778v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"i-can-t-believe-there-s-no-images-learning","repo_url":"https://github.com/allenai/close","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-entailment","task_name":"Visual Entailment"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2211.09778","atlas_url":"https://app.syntology.ai/?focus=2211.09778","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.09778"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/close","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"091595d4fe6a6678","entry":"generate_coco_style_prompt","repo":"allenai/close","repo_kind":"official","path":"close/experiments/generate_stylistic_captioning.py","file_url":"https://github.com/allenai/close/blob/HEAD/close/experiments/generate_stylistic_captioning.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"091595d4fe6a6678"}},{"code_sha256_prefix":"e8bca3ee0039d434","entry":"generate_ego_centric_caption","repo":"allenai/close","repo_kind":"official","path":"close/experiments/generate_stylistic_captioning.py","file_url":"https://github.com/allenai/close/blob/HEAD/close/experiments/generate_stylistic_captioning.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e8bca3ee0039d434"}},{"code_sha256_prefix":"e8fca5d13a0be178","entry":"generate_harry_potter_prompt","repo":"allenai/close","repo_kind":"official","path":"close/experiments/generate_stylistic_captioning.py","file_url":"https://github.com/allenai/close/blob/HEAD/close/experiments/generate_stylistic_captioning.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e8fca5d13a0be178"}},{"code_sha256_prefix":"dd4fbc83b18bbb10","entry":"processDigitArticle","repo":"allenai/close","repo_kind":"official","path":"close/eval/vqa_eval.py","file_url":"https://github.com/allenai/close/blob/HEAD/close/eval/vqa_eval.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dd4fbc83b18bbb10"}},{"code_sha256_prefix":"54ecd7319b5ad5b5","entry":"processPunctuation","repo":"allenai/close","repo_kind":"official","path":"close/eval/vqa_eval.py","file_url":"https://github.com/allenai/close/blob/HEAD/close/eval/vqa_eval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"54ecd7319b5ad5b5"}},{"code_sha256_prefix":"d346aeb842401f90","entry":"vqa_preprocess","repo":"allenai/close","repo_kind":"official","path":"close/eval/vqa_eval.py","file_url":"https://github.com/allenai/close/blob/HEAD/close/eval/vqa_eval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d346aeb842401f90"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}