{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/languagebind-extending-video-language","title":"LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment","arxiv_id":"2310.01852","date":"2023-10-03","proceeding":null,"authors":["Bin Zhu","Bin Lin","Munan Ning","Yang Yan","Jiaxi Cui","Hongfa Wang","Yatian Pang","Wenhao Jiang","Junwu Zhang","Zongwei Li","Wancai Zhang","Zhifeng Li","Wei Liu","Li Yuan"],"abstract":"The video-language (VL) pretraining has achieved remarkable improvement in multiple downstream tasks. However, the current VL pretraining framework is hard to extend to multiple modalities (N modalities, N>=3) beyond vision and language. We thus propose LanguageBind, taking the language as the bind across different modalities because the language modality is well-explored and contains rich semantics. Specifically, we freeze the language encoder acquired by VL pretraining, then train encoders for other modalities with contrastive learning. As a result, all modalities are mapped to a shared feature space, implementing multi-modal semantic alignment. While LanguageBind ensures that we can extend VL modalities to N modalities, we also need a high-quality dataset with alignment data pairs centered on language. We thus propose VIDAL-10M with Video, Infrared, Depth, Audio and their corresponding Language, naming as VIDAL-10M. In our VIDAL-10M, all videos are from short video platforms with complete semantics rather than truncated segments from long videos, and all the video, depth, infrared, and audio modalities are aligned to their textual descriptions. LanguageBind has achieved superior performance on a wide range of 15 benchmarks covering video, audio, depth, and infrared. Moreover, multiple experiments have provided evidence for the effectiveness of LanguageBind in achieving indirect alignment and complementarity among diverse modalities. Code address: https://github.com/PKU-YuanGroup/LanguageBind","url_abs":"https://arxiv.org/abs/2310.01852v7","url_pdf":"https://arxiv.org/pdf/2310.01852v7.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"languagebind-extending-video-language","repo_url":"https://github.com/pku-yuangroup/languagebind","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"languagebind-extending-video-language","repo_url":"https://github.com/PKU-YuanGroup/LLMBind","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"languagebind-extending-video-language","repo_url":"https://github.com/PKU-YuanGroup/MoE-LLaVA","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"languagebind-extending-video-language","repo_url":"https://github.com/PKU-YuanGroup/Video-LLaVA","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"languagebind-extending-video-language","repo_url":"https://github.com/pku-yuangroup/video-bench","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"languagebind-extending-video-language","repo_url":"https://github.com/zhihaozhang97/ru-ai","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"audio-classification","task_name":"Audio Classification"},{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"multimodal-deep-learning","task_name":"Multimodal Deep Learning"},{"task_slug":"scene-classification-unified-classes","task_name":"Scene Classification (unified classes)"},{"task_slug":"temporal-relation-extraction","task_name":"Temporal Relation Extraction"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"video-text-retrieval","task_name":"Video-Text Retrieval"},{"task_slug":"zero-shot-action-recognition","task_name":"Zero-Shot Action Recognition"},{"task_slug":"zero-shot-environment-sound-classification","task_name":"Zero-Shot Environment Sound Classification"},{"task_slug":"zero-shot-video-retrieval","task_name":"Zero-Shot Video Retrieval"},{"task_slug":"zero-shot-audio-classification","task_name":"Zero-shot Audio Classification"},{"task_slug":"zero-shot-classification-unified-classes","task_name":"Zero-shot Classification (unified classes)"},{"task_slug":"zero-shot-scene-classification-unified","task_name":"Zero-shot Scene Classification (unified classes)"},{"task_slug":"zero-shot-text-retrieval","task_name":"Zero-shot Text Retrieval"},{"task_slug":"zero-shot-text-to-audio-retrieval","task_name":"Zero-shot Text to Audio Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/temporal-relation-extraction-on-vinoground","task":"Temporal Relation Extraction","dataset":"Vinoground","model":"LanguageBind","rank_in_archive_order":23,"of":24,"metrics":{"Group Score":"1.2","Text Score":"10.6","Video Score":"5"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-action-recognition-on-kinetics","task":"Zero-Shot Action Recognition","dataset":"Kinetics","model":"LanguageBind","rank_in_archive_order":10,"of":20,"metrics":{"Top-1 Accuracy":"64.1","Top-5 Accuracy":"85.7"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-activitynet","task":"Zero-Shot Video Retrieval","dataset":"ActivityNet","model":"LanguageBind(ViT-H/14)","rank_in_archive_order":6,"of":12,"metrics":{"text-to-video R@1":"41.0","text-to-video R@10":"80.0","text-to-video R@5":"68.4","video-to-text R@1":"39.1","video-to-text R@10":"81.1","video-to-text R@5":"69.8"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-activitynet","task":"Zero-Shot Video Retrieval","dataset":"ActivityNet","model":"LanguageBind(ViT-L/14)","rank_in_archive_order":7,"of":12,"metrics":{"text-to-video R@1":"38.4","text-to-video R@10":"77.9","text-to-video R@5":"66.6","video-to-text R@1":"35.7","video-to-text R@10":"77.8","video-to-text R@5":"65.8"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-didemo","task":"Zero-Shot Video Retrieval","dataset":"DiDeMo","model":"LanguageBind(ViT-H/14)","rank_in_archive_order":9,"of":26,"metrics":{"text-to-video Median Rank":"2","text-to-video R@1":"39.9","text-to-video R@10":"74.6","text-to-video R@5":"66.1","video-to-text R@1":"39.8","video-to-text R@10":"76.2","video-to-text R@5":"67.8"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-didemo","task":"Zero-Shot Video Retrieval","dataset":"DiDeMo","model":"LanguageBind(ViT-L/14)","rank_in_archive_order":10,"of":26,"metrics":{"text-to-video Median Rank":"2.0","text-to-video R@1":"39.7","text-to-video R@10":"73.8","text-to-video R@5":"65.5","video-to-text R@1":"38.4","video-to-text R@10":"77.9","video-to-text R@5":"66.6"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msr-vtt","task":"Zero-Shot Video Retrieval","dataset":"MSR-VTT","model":"LanguageBind(ViT-H/14)","rank_in_archive_order":9,"of":41,"metrics":{"text-to-video Median Rank":"2","text-to-video R@1":"44.8","text-to-video R@10":"78.7","text-to-video R@5":"70.0","video-to-text Median Rank":"2.","video-to-text R@1":"40.9","video-to-text R@10":"75.7","video-to-text R@5":"66.4"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msr-vtt","task":"Zero-Shot Video Retrieval","dataset":"MSR-VTT","model":"LanguageBind(ViT-L/14)","rank_in_archive_order":10,"of":41,"metrics":{"text-to-video Median Rank":"2.0","text-to-video R@1":"42.8","text-to-video R@10":"76.0","text-to-video R@5":"67.5","video-to-text Median Rank":"3.0","video-to-text R@1":"38.3","video-to-text R@10":"77.8","video-to-text R@5":"65.8"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msvd","task":"Zero-Shot Video Retrieval","dataset":"MSVD","model":"LanguageBind(ViT-L/14)","rank_in_archive_order":4,"of":14,"metrics":{"text-to-video Median Rank":"1.0","text-to-video R@1":"54.1","text-to-video R@10":"88.1","text-to-video R@5":"81.1","video-to-text Median Rank":"1.0","video-to-text R@1":"69.7","video-to-text R@10":"97.9","video-to-text R@5":"91.8"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msvd","task":"Zero-Shot Video Retrieval","dataset":"MSVD","model":"LanguageBind(ViT-H/14)","rank_in_archive_order":5,"of":14,"metrics":{"text-to-video Median Rank":"1","text-to-video R@1":"53.9","text-to-video R@10":"87.8","text-to-video R@5":"80.4","video-to-text Median Rank":"1","video-to-text R@1":"72.0","video-to-text R@10":"96.3","video-to-text R@5":"91.4"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.01852","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.01852"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/PKU-YuanGroup/LanguageBind","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PKU-YuanGroup/Video-LLaVA","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pku-yuangroup/video-bench","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PKU-YuanGroup/MoE-LLaVA","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhihaozhang97/ru-ai","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pku-yuangroup/languagebind","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PKU-YuanGroup/LLMBind","reach":null}],"summary":{"ran_fixture":1,"ran":4,"ran_honours":2,"unverified":7},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1},"listed":{"samples":9,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"751b1c8ccbe160c4","entry":"cast_if_src_dtype","repo":"zhihaozhang97/ru-ai","repo_kind":"listed","path":"imagebind/models/helpers.py","file_url":"https://github.com/zhihaozhang97/ru-ai/blob/HEAD/imagebind/models/helpers.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"751b1c8ccbe160c4"}},{"code_sha256_prefix":"432c312da80537d6","entry":"get_clip_timepoints","repo":"zhihaozhang97/ru-ai","repo_kind":"listed","path":"imagebind/data.py","file_url":"https://github.com/zhihaozhang97/ru-ai/blob/HEAD/imagebind/data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"432c312da80537d6"}},{"code_sha256_prefix":"ac6a026c5d4ff47b","entry":"get_depth_transform","repo":"PKU-YuanGroup/LanguageBind","repo_kind":"official","path":"languagebind/depth/processing_depth.py","file_url":"https://github.com/PKU-YuanGroup/LanguageBind/blob/HEAD/languagebind/depth/processing_depth.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac6a026c5d4ff47b"}},{"code_sha256_prefix":"35a743e50e17e66d","entry":"get_sinusoid_encoding_table","repo":"zhihaozhang97/ru-ai","repo_kind":"listed","path":"imagebind/models/multimodal_preprocessors.py","file_url":"https://github.com/zhihaozhang97/ru-ai/blob/HEAD/imagebind/models/multimodal_preprocessors.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"35a743e50e17e66d"}},{"code_sha256_prefix":"20a58a8c04ccda43","entry":"int16_to_float32_torch","repo":"PKU-YuanGroup/LanguageBind","repo_kind":"official","path":"languagebind/audio/processing_audio.py","file_url":"https://github.com/PKU-YuanGroup/LanguageBind/blob/HEAD/languagebind/audio/processing_audio.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20a58a8c04ccda43"}},{"code_sha256_prefix":"5530423121ac4757","entry":"opencv_loader","repo":"PKU-YuanGroup/LanguageBind","repo_kind":"official","path":"languagebind/depth/processing_depth.py","file_url":"https://github.com/PKU-YuanGroup/LanguageBind/blob/HEAD/languagebind/depth/processing_depth.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5530423121ac4757"}},{"code_sha256_prefix":"833effea5806a36e","entry":"stack_dict","repo":"PKU-YuanGroup/LanguageBind","repo_kind":"official","path":"model/languagebind.py","file_url":"https://github.com/PKU-YuanGroup/LanguageBind/blob/HEAD/model/languagebind.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"833effea5806a36e"}},{"code_sha256_prefix":"81763ce5edd977e6","entry":"combine_parts","repo":"zhihaozhang97/ru-ai","repo_kind":"listed","path":"download_all.py","file_url":"https://github.com/zhihaozhang97/ru-ai/blob/HEAD/download_all.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81763ce5edd977e6"}},{"code_sha256_prefix":"67710298891367de","entry":"get_deposition_files","repo":"zhihaozhang97/ru-ai","repo_kind":"listed","path":"download_all.py","file_url":"https://github.com/zhihaozhang97/ru-ai/blob/HEAD/download_all.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"67710298891367de"}},{"code_sha256_prefix":"bee2c9fa238a8b73","entry":"imagebind_huge","repo":"zhihaozhang97/ru-ai","repo_kind":"listed","path":"imagebind/models/imagebind_model.py","file_url":"https://github.com/zhihaozhang97/ru-ai/blob/HEAD/imagebind/models/imagebind_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bee2c9fa238a8b73"}},{"code_sha256_prefix":"71227efe6a7df1d2","entry":"interpolate_pos_encoding","repo":"zhihaozhang97/ru-ai","repo_kind":"listed","path":"imagebind/models/multimodal_preprocessors.py","file_url":"https://github.com/zhihaozhang97/ru-ai/blob/HEAD/imagebind/models/multimodal_preprocessors.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71227efe6a7df1d2"}},{"code_sha256_prefix":"ddd2e52e6b0956b5","entry":"interpolate_pos_encoding_2d","repo":"zhihaozhang97/ru-ai","repo_kind":"listed","path":"imagebind/models/multimodal_preprocessors.py","file_url":"https://github.com/zhihaozhang97/ru-ai/blob/HEAD/imagebind/models/multimodal_preprocessors.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ddd2e52e6b0956b5"}},{"code_sha256_prefix":"3b8a9225053af887","entry":"load_and_transform_vision_data","repo":"zhihaozhang97/ru-ai","repo_kind":"listed","path":"imagebind/data.py","file_url":"https://github.com/zhihaozhang97/ru-ai/blob/HEAD/imagebind/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b8a9225053af887"}},{"code_sha256_prefix":"852e87ab903330a7","entry":"make_list_of_images","repo":"PKU-YuanGroup/LanguageBind","repo_kind":"official","path":"languagebind/audio/processing_audio.py","file_url":"https://github.com/PKU-YuanGroup/LanguageBind/blob/HEAD/languagebind/audio/processing_audio.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"852e87ab903330a7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}