{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/frozen-transformers-in-language-models-are","title":"Frozen Transformers in Language Models Are Effective Visual Encoder Layers","arxiv_id":"2310.12973","date":"2023-10-19","proceeding":null,"authors":["Ziqi Pang","Ziyang Xie","Yunze Man","Yu-Xiong Wang"],"abstract":"This paper reveals that large language models (LLMs), despite being trained solely on textual data, are surprisingly strong encoders for purely visual tasks in the absence of language. Even more intriguingly, this can be achieved by a simple yet previously overlooked strategy -- employing a frozen transformer block from pre-trained LLMs as a constituent encoder layer to directly process visual tokens. Our work pushes the boundaries of leveraging LLMs for computer vision tasks, significantly departing from conventional practices that typically necessitate a multi-modal vision-language setup with associated language prompts, inputs, or outputs. We demonstrate that our approach consistently enhances performance across a diverse range of tasks, encompassing pure 2D and 3D visual recognition tasks (e.g., image and point cloud classification), temporal modeling tasks (e.g., action recognition), non-semantic tasks (e.g., motion forecasting), and multi-modal tasks (e.g., 2D/3D visual question answering and image-text retrieval). Such improvements are a general phenomenon, applicable to various types of LLMs (e.g., LLaMA and OPT) and different LLM transformer blocks. We additionally propose the information filtering hypothesis to explain the effectiveness of pre-trained LLMs in visual encoding -- the pre-trained LLM transformer blocks discern informative visual tokens and further amplify their effect. This hypothesis is empirically supported by the observation that the feature activation, after training with LLM transformer blocks, exhibits a stronger focus on relevant regions. We hope that our work inspires new perspectives on utilizing LLMs and deepening our understanding of their underlying mechanisms. Code is available at https://github.com/ziqipang/LM4VisualEncoding.","url_abs":"https://arxiv.org/abs/2310.12973v2","url_pdf":"https://arxiv.org/pdf/2310.12973v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"frozen-transformers-in-language-models-are","repo_url":"https://github.com/ziqipang/lm4visualencoding","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"frozen-transformers-in-language-models-are","repo_url":"https://github.com/zhixinlai/llmboostmedical","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"image-text-retrieval","task_name":"Image-text Retrieval"},{"task_slug":"motion-forecasting","task_name":"Motion Forecasting"},{"task_slug":"point-cloud-classification","task_name":"Point Cloud Classification"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/question-answering-on-sqa3d","task":"Question Answering","dataset":"SQA3D","model":"LM4VisualEncoding","rank_in_archive_order":4,"of":7,"metrics":{"AnswerExactMatch (Question Answering)":"48.09"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.12973","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.12973"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ziqipang/lm4visualencoding","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhixinlai/llmboostmedical","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_fixture":2,"ran_honours":1,"ran_violates":1,"unverified":8},"by_repo_kind":{"official":{"samples":16,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e805e183bad0f027","entry":"add_weight_decay","repo":"ziqipang/lm4visualencoding","repo_kind":"official","path":"image_classification/optimizer_utils.py","file_url":"https://github.com/ziqipang/lm4visualencoding/blob/HEAD/image_classification/optimizer_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e805e183bad0f027"}},{"code_sha256_prefix":"b47d48e431b34acd","entry":"apply_rotary_emb","repo":"ziqipang/lm4visualencoding","repo_kind":"official","path":"image_classification/models/llama.py","file_url":"https://github.com/ziqipang/lm4visualencoding/blob/HEAD/image_classification/models/llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b47d48e431b34acd"}},{"code_sha256_prefix":"a9e48df75d2e823e","entry":"get_mce_from_accuracy","repo":"ziqipang/lm4visualencoding","repo_kind":"official","path":"image_classification/utils.py","file_url":"https://github.com/ziqipang/lm4visualencoding/blob/HEAD/image_classification/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a9e48df75d2e823e"}},{"code_sha256_prefix":"da651e3979a18f84","entry":"get_sinusoid_encoding_table","repo":"ziqipang/lm4visualencoding","repo_kind":"official","path":"video_understanding/modeling_finetune.py","file_url":"https://github.com/ziqipang/lm4visualencoding/blob/HEAD/video_understanding/modeling_finetune.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da651e3979a18f84"}},{"code_sha256_prefix":"2cd1e282432041c8","entry":"optimizer_kwargs","repo":"ziqipang/lm4visualencoding","repo_kind":"official","path":"image_classification/optimizer_utils.py","file_url":"https://github.com/ziqipang/lm4visualencoding/blob/HEAD/image_classification/optimizer_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2cd1e282432041c8"}},{"code_sha256_prefix":"14a84c2cbfebc413","entry":"precompute_freqs_cis","repo":"ziqipang/lm4visualencoding","repo_kind":"official","path":"image_classification/models/llama.py","file_url":"https://github.com/ziqipang/lm4visualencoding/blob/HEAD/image_classification/models/llama.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"14a84c2cbfebc413"}},{"code_sha256_prefix":"70bf6ebaafd266c4","entry":"reshape_for_broadcast","repo":"ziqipang/lm4visualencoding","repo_kind":"official","path":"image_classification/models/llama.py","file_url":"https://github.com/ziqipang/lm4visualencoding/blob/HEAD/image_classification/models/llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"70bf6ebaafd266c4"}},{"code_sha256_prefix":"32208d3133397a32","entry":"unnormalize_image","repo":"ziqipang/lm4visualencoding","repo_kind":"official","path":"image_classification/utils.py","file_url":"https://github.com/ziqipang/lm4visualencoding/blob/HEAD/image_classification/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"32208d3133397a32"}},{"code_sha256_prefix":"58ab30e8c22165b6","entry":"PGDAttack","repo":"ziqipang/lm4visualencoding","repo_kind":"official","path":"image_classification/engine.py","file_url":"https://github.com/ziqipang/lm4visualencoding/blob/HEAD/image_classification/engine.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"58ab30e8c22165b6"}},{"code_sha256_prefix":"8b43480472330eaf","entry":"build_dataset","repo":"ziqipang/lm4visualencoding","repo_kind":"official","path":"image_classification/datasets.py","file_url":"https://github.com/ziqipang/lm4visualencoding/blob/HEAD/image_classification/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b43480472330eaf"}},{"code_sha256_prefix":"3bc74137ab36aa79","entry":"build_transform","repo":"ziqipang/lm4visualencoding","repo_kind":"official","path":"image_classification/datasets.py","file_url":"https://github.com/ziqipang/lm4visualencoding/blob/HEAD/image_classification/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3bc74137ab36aa79"}},{"code_sha256_prefix":"8a93e041134b597a","entry":"clamp","repo":"ziqipang/lm4visualencoding","repo_kind":"official","path":"image_classification/engine.py","file_url":"https://github.com/ziqipang/lm4visualencoding/blob/HEAD/image_classification/engine.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a93e041134b597a"}},{"code_sha256_prefix":"29416043c7035e7c","entry":"get_loss_scale_for_deepspeed","repo":"ziqipang/lm4visualencoding","repo_kind":"official","path":"video_understanding/engine_for_finetuning.py","file_url":"https://github.com/ziqipang/lm4visualencoding/blob/HEAD/video_understanding/engine_for_finetuning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"29416043c7035e7c"}},{"code_sha256_prefix":"a57f32dae793402d","entry":"patch_level_aug","repo":"ziqipang/lm4visualencoding","repo_kind":"official","path":"image_classification/engine.py","file_url":"https://github.com/ziqipang/lm4visualencoding/blob/HEAD/image_classification/engine.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a57f32dae793402d"}},{"code_sha256_prefix":"14b3b13b2e7ee844","entry":"tensor_normalize","repo":"ziqipang/lm4visualencoding","repo_kind":"official","path":"video_understanding/kinetics.py","file_url":"https://github.com/ziqipang/lm4visualencoding/blob/HEAD/video_understanding/kinetics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"14b3b13b2e7ee844"}},{"code_sha256_prefix":"c7976ea27edc377a","entry":"train_class_batch","repo":"ziqipang/lm4visualencoding","repo_kind":"official","path":"video_understanding/engine_for_finetuning.py","file_url":"https://github.com/ziqipang/lm4visualencoding/blob/HEAD/video_understanding/engine_for_finetuning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7976ea27edc377a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}