{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/linearly-mapping-from-image-to-text-space","title":"Linearly Mapping from Image to Text Space","arxiv_id":"2209.15162","date":"2022-09-30","proceeding":null,"authors":["Jack Merullo","Louis Castricato","Carsten Eickhoff","Ellie Pavlick"],"abstract":"The extent to which text-only language models (LMs) learn to represent features of the non-linguistic world is an open question. Prior work has shown that pretrained LMs can be taught to caption images when a vision model's parameters are optimized to encode images in the language space. We test a stronger hypothesis: that the conceptual representations learned by frozen text-only models and vision-only models are similar enough that this can be achieved with a linear map. We show that the image representations from vision models can be transferred as continuous prompts to frozen LMs by training only a single linear projection. Using these to prompt the LM achieves competitive performance on captioning and visual question answering tasks compared to models that tune both the image encoder and text decoder (such as the MAGMA model). We compare three image encoders with increasing amounts of linguistic supervision seen during pretraining: BEIT (no linguistic information), NF-ResNET (lexical category information), and CLIP (full natural language descriptions). We find that all three encoders perform equally well at transferring visual property information to the language model (e.g., whether an animal is large or small), but that image encoders pretrained with linguistic supervision more saliently encode category information (e.g., distinguishing hippo vs. elephant) and thus perform significantly better on benchmark language-and-vision tasks. Our results indicate that LMs encode conceptual information structurally similarly to vision-based models, even those that are solely trained on images. Code is available here: https://github.com/jmerullo/limber","url_abs":"https://arxiv.org/abs/2209.15162v3","url_pdf":"https://arxiv.org/pdf/2209.15162v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"linearly-mapping-from-image-to-text-space","repo_url":"https://github.com/jmerullo/limber","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"linearly-mapping-from-image-to-text-space","repo_url":"https://github.com/bytetriper/lm4lv","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"image-to-text","task_name":"Image to text"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"open-question","task_name":"Open-Ended Question Answering"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2209.15162","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.15162"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/jmerullo/limber","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bytetriper/lm4lv","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5,"unverified":9},"by_repo_kind":{"official":{"samples":14,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ace12fad87592626","entry":"load_config","repo":"jmerullo/limber","repo_kind":"official","path":"config.py","file_url":"https://github.com/jmerullo/limber/blob/HEAD/config.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ace12fad87592626"}},{"code_sha256_prefix":"e82e4764f914492f","entry":"pad_to_size","repo":"jmerullo/limber","repo_kind":"official","path":"transforms.py","file_url":"https://github.com/jmerullo/limber/blob/HEAD/transforms.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e82e4764f914492f"}},{"code_sha256_prefix":"d3af6b05f019b479","entry":"pad_to_size_tensor","repo":"jmerullo/limber","repo_kind":"official","path":"transforms.py","file_url":"https://github.com/jmerullo/limber/blob/HEAD/transforms.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d3af6b05f019b479"}},{"code_sha256_prefix":"ba246ce8c9917aba","entry":"remove_tokens_after_eos","repo":"jmerullo/limber","repo_kind":"official","path":"sampling.py","file_url":"https://github.com/jmerullo/limber/blob/HEAD/sampling.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba246ce8c9917aba"}},{"code_sha256_prefix":"aa8cc3e8f125148a","entry":"top_k_filter","repo":"jmerullo/limber","repo_kind":"official","path":"sampling.py","file_url":"https://github.com/jmerullo/limber/blob/HEAD/sampling.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa8cc3e8f125148a"}},{"code_sha256_prefix":"43c05e1f4efea52f","entry":"beit_feature_extractor","repo":"jmerullo/limber","repo_kind":"official","path":"transforms.py","file_url":"https://github.com/jmerullo/limber/blob/HEAD/transforms.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"43c05e1f4efea52f"}},{"code_sha256_prefix":"435a97d7491369dc","entry":"collate_fn","repo":"jmerullo/limber","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/jmerullo/limber/blob/HEAD/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"435a97d7491369dc"}},{"code_sha256_prefix":"b93589f28f3dd515","entry":"configure_param_groups","repo":"jmerullo/limber","repo_kind":"official","path":"config_params.py","file_url":"https://github.com/jmerullo/limber/blob/HEAD/config_params.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b93589f28f3dd515"}},{"code_sha256_prefix":"32d95b9de0cbb2de","entry":"full_nfresnet50","repo":"jmerullo/limber","repo_kind":"official","path":"image_encoders.py","file_url":"https://github.com/jmerullo/limber/blob/HEAD/image_encoders.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"32d95b9de0cbb2de"}},{"code_sha256_prefix":"f2aaccc3c641089b","entry":"get_params_for_weight_decay_optimization","repo":"jmerullo/limber","repo_kind":"official","path":"config_params.py","file_url":"https://github.com/jmerullo/limber/blob/HEAD/config_params.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2aaccc3c641089b"}},{"code_sha256_prefix":"1fe016f046db89bd","entry":"get_tokenizer","repo":"jmerullo/limber","repo_kind":"official","path":"limber_gptj.py","file_url":"https://github.com/jmerullo/limber/blob/HEAD/limber_gptj.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1fe016f046db89bd"}},{"code_sha256_prefix":"26c48ad88add4d84","entry":"load_json","repo":"jmerullo/limber","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/jmerullo/limber/blob/HEAD/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26c48ad88add4d84"}},{"code_sha256_prefix":"81a63795029ff25d","entry":"nfresnet50","repo":"jmerullo/limber","repo_kind":"official","path":"image_encoders.py","file_url":"https://github.com/jmerullo/limber/blob/HEAD/image_encoders.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81a63795029ff25d"}},{"code_sha256_prefix":"a4d1b53e4f72d453","entry":"read_img_captions","repo":"jmerullo/limber","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/jmerullo/limber/blob/HEAD/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a4d1b53e4f72d453"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}