{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/instraug-automatic-instruction-augmentation","title":"Towards Robust Instruction Tuning on Multimodal Large Language Models","arxiv_id":"2402.14492","date":"2024-02-22","proceeding":null,"authors":["Wei Han","Hui Chen","Soujanya Poria"],"abstract":"Fine-tuning large language models (LLMs) on multi-task instruction-following data has been proven to be a powerful learning paradigm for improving their zero-shot capabilities on new tasks. Recent works about high-quality instruction-following data generation and selection require amounts of human labor to conceive model-understandable instructions for the given tasks and carefully filter the LLM-generated data. In this work, we introduce an automatic instruction augmentation method named INSTRAUG in multimodal tasks. It starts from a handful of basic and straightforward meta instructions but can expand an instruction-following dataset by 30 times. Results on two popular multimodal instructionfollowing benchmarks MULTIINSTRUCT and InstructBLIP show that INSTRAUG can significantly improve the alignment of multimodal large language models (MLLMs) across 12 multimodal tasks, which is even equivalent to the benefits of scaling up training data multiple times.","url_abs":"https://arxiv.org/abs/2402.14492v2","url_pdf":"https://arxiv.org/pdf/2402.14492v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"instraug-automatic-instruction-augmentation","repo_url":"https://github.com/declare-lab/instraug","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.14492","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.14492"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/declare-lab/instraug","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"ran":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"357d3a4f0be679d9","entry":"apply_half","repo":"declare-lab/instraug","repo_kind":"official","path":"models/OFA/evaluate.py","file_url":"https://github.com/declare-lab/instraug/blob/HEAD/models/OFA/evaluate.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"357d3a4f0be679d9"}},{"code_sha256_prefix":"054546da8bfefb43","entry":"build_instruction","repo":"declare-lab/instraug","repo_kind":"official","path":"AutoScaling/instruction_templates.py","file_url":"https://github.com/declare-lab/instraug/blob/HEAD/AutoScaling/instruction_templates.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"054546da8bfefb43"}},{"code_sha256_prefix":"54dfc0cde2d896f2","entry":"convert_name","repo":"declare-lab/instraug","repo_kind":"official","path":"AutoScaling/build_dataset.py","file_url":"https://github.com/declare-lab/instraug/blob/HEAD/AutoScaling/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54dfc0cde2d896f2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}