{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/parrot-multilingual-visual-instruction-tuning","title":"Parrot: Multilingual Visual Instruction Tuning","arxiv_id":"2406.02539","date":"2024-06-04","proceeding":null,"authors":["Hai-Long Sun","Da-Wei Zhou","Yang Li","Shiyin Lu","Chao Yi","Qing-Guo Chen","Zhao Xu","Weihua Luo","Kaifu Zhang","De-Chuan Zhan","Han-Jia Ye"],"abstract":"The rapid development of Multimodal Large Language Models (MLLMs) like GPT-4V has marked a significant step towards artificial general intelligence. Existing methods mainly focus on aligning vision encoders with LLMs through supervised fine-tuning (SFT) to endow LLMs with multimodal abilities, making MLLMs' inherent ability to react to multiple languages progressively deteriorate as the training process evolves. We empirically find that the imbalanced SFT datasets, primarily composed of English-centric image-text pairs, lead to significantly reduced performance in non-English languages. This is due to the failure of aligning the vision encoder and LLM with multilingual tokens during the SFT process. In this paper, we introduce Parrot, a novel method that utilizes textual guidance to drive visual token alignment at the language level. Parrot makes the visual tokens condition on diverse language inputs and uses Mixture-of-Experts (MoE) to promote the alignment of multilingual tokens. Specifically, to enhance non-English visual tokens alignment, we compute the cross-attention using the initial visual features and textual embeddings, the result of which is then fed into the MoE router to select the most relevant experts. The selected experts subsequently convert the initial visual tokens into language-specific visual tokens. Moreover, considering the current lack of benchmarks for evaluating multilingual capabilities within the field, we collect and make available a Massive Multilingual Multimodal Benchmark which includes 6 languages, 15 categories, and 12,000 questions, named as MMMB. Our method not only demonstrates state-of-the-art performance on multilingual MMBench and MMMB, but also excels across a broad range of multimodal tasks. Both the source code and the training dataset of Parrot will be made publicly available. Code is available at: https://github.com/AIDC-AI/Parrot.","url_abs":"https://arxiv.org/abs/2406.02539v2","url_pdf":"https://arxiv.org/pdf/2406.02539v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"parrot-multilingual-visual-instruction-tuning","repo_url":"https://github.com/aidc-ai/parrot","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"parrot-multilingual-visual-instruction-tuning","repo_url":"https://github.com/aidc-ai/ovis","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"moe","method_name":"MoE"},{"method_slug":"parrot","method_name":"Parrot"},{"method_slug":"sft","method_name":"SFT"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.02539","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.02539"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/AIDC-AI/Parrot","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aidc-ai/parrot","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aidc-ai/ovis","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"ran_fixture":2,"unverified":4},"by_repo_kind":{"official":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ef4db0da176c674c","entry":"build_vision_projector","repo":"AIDC-AI/Parrot","repo_kind":"official","path":"parrot/model/multimodal_projector/builder.py","file_url":"https://github.com/AIDC-AI/Parrot/blob/HEAD/parrot/model/multimodal_projector/builder.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ef4db0da176c674c"}},{"code_sha256_prefix":"10893c4608c08075","entry":"split_to_even_chunks","repo":"AIDC-AI/Parrot","repo_kind":"official","path":"parrot/train/parrot_trainer.py","file_url":"https://github.com/AIDC-AI/Parrot/blob/HEAD/parrot/train/parrot_trainer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"10893c4608c08075"}},{"code_sha256_prefix":"7606525af238fb64","entry":"unpad_image","repo":"AIDC-AI/Parrot","repo_kind":"official","path":"parrot/model/parrot_arch.py","file_url":"https://github.com/AIDC-AI/Parrot/blob/HEAD/parrot/model/parrot_arch.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7606525af238fb64"}},{"code_sha256_prefix":"fd2cc4319a956bc2","entry":"ParrotMetaForCausalLM","repo":"aidc-ai/parrot","repo_kind":"official","path":"parrot/model/parrot_arch.py","file_url":"https://github.com/aidc-ai/parrot/blob/HEAD/parrot/model/parrot_arch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fd2cc4319a956bc2"}},{"code_sha256_prefix":"aeaec2cd790d9027","entry":"build_vision_tower","repo":"AIDC-AI/Parrot","repo_kind":"official","path":"parrot/model/multimodal_encoder/builder.py","file_url":"https://github.com/AIDC-AI/Parrot/blob/HEAD/parrot/model/multimodal_encoder/builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aeaec2cd790d9027"}},{"code_sha256_prefix":"bb35e3ac741bb2c9","entry":"get_mm_adapter_state_maybe_zero_3","repo":"AIDC-AI/Parrot","repo_kind":"official","path":"parrot/train/parrot_trainer.py","file_url":"https://github.com/AIDC-AI/Parrot/blob/HEAD/parrot/train/parrot_trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bb35e3ac741bb2c9"}},{"code_sha256_prefix":"735025744c1ab0cf","entry":"maybe_zero_3","repo":"AIDC-AI/Parrot","repo_kind":"official","path":"parrot/train/parrot_trainer.py","file_url":"https://github.com/AIDC-AI/Parrot/blob/HEAD/parrot/train/parrot_trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"735025744c1ab0cf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}