{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/your-vision-language-model-itself-is-a-strong","title":"Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection","arxiv_id":"2402.12501","date":"2024-02-19","proceeding":null,"authors":["Ruibo Chen","Yihan Wu","Lichang Chen","Guodong Liu","Qi He","Tianyi Xiong","Chenxi Liu","Junfeng Guo","Heng Huang"],"abstract":"Data selection in instruction tuning emerges as a pivotal process for acquiring high-quality data and training instruction-following large language models (LLMs), but it is still a new and unexplored research area for vision-language models (VLMs). Existing data selection approaches on LLMs either rely on single unreliable scores, or use downstream tasks for selection, which is time-consuming and can lead to potential over-fitting on the chosen evaluation datasets. To address this challenge, we introduce a novel dataset selection method, Self-Filter, that utilizes the VLM itself as a filter. This approach is inspired by the observation that VLMs benefit from training with the most challenging instructions. Self-Filter operates in two stages. In the first stage, we devise a scoring network to evaluate the difficulty of training instructions, which is co-trained with the VLM. In the second stage, we use the trained score net to measure the difficulty of each instruction, select the most challenging samples, and penalize similar samples to encourage diversity. Comprehensive experiments on LLaVA and MiniGPT-4 show that Self-Filter can reach better results compared to full data settings with merely about 15% samples, and can achieve superior performance against competitive baselines.","url_abs":"https://arxiv.org/abs/2402.12501v1","url_pdf":"https://arxiv.org/pdf/2402.12501v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"your-vision-language-model-itself-is-a-strong","repo_url":"https://github.com/rayruibochen/self-filter","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"AGPL-3.0"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.12501","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.12501"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rayruibochen/self-filter","reach":{"status":"ok","spdx":"AGPL-3.0"}}],"summary":{"ran":5,"ran_fixture":1,"unverified":7},"by_repo_kind":{"official":{"samples":13,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"a9f557656b27e601","entry":"get_input_text","repo":"rayruibochen/self-filter","repo_kind":"official","path":"data_process/preprocess.py","file_url":"https://github.com/rayruibochen/self-filter/blob/HEAD/data_process/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a9f557656b27e601"}},{"code_sha256_prefix":"c6d149d9ee4d25ca","entry":"get_text_instruction","repo":"rayruibochen/self-filter","repo_kind":"official","path":"data_process/preprocess.py","file_url":"https://github.com/rayruibochen/self-filter/blob/HEAD/data_process/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c6d149d9ee4d25ca"}},{"code_sha256_prefix":"17ea60a8bf7dc6fd","entry":"load_scores","repo":"rayruibochen/self-filter","repo_kind":"official","path":"self_filter/stage2.py","file_url":"https://github.com/rayruibochen/self-filter/blob/HEAD/self_filter/stage2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"17ea60a8bf7dc6fd"}},{"code_sha256_prefix":"b6851106077336dd","entry":"norm_scores","repo":"rayruibochen/self-filter","repo_kind":"official","path":"data_process/analysis_gpt.py","file_url":"https://github.com/rayruibochen/self-filter/blob/HEAD/data_process/analysis_gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b6851106077336dd"}},{"code_sha256_prefix":"10893c4608c08075","entry":"split_to_even_chunks","repo":"rayruibochen/self-filter","repo_kind":"official","path":"self_filter/self_filter_trainer.py","file_url":"https://github.com/rayruibochen/self-filter/blob/HEAD/self_filter/self_filter_trainer.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"10893c4608c08075"}},{"code_sha256_prefix":"562cce8b27dea629","entry":"warp_inst","repo":"rayruibochen/self-filter","repo_kind":"official","path":"data_process/query_gpt_text_models.py","file_url":"https://github.com/rayruibochen/self-filter/blob/HEAD/data_process/query_gpt_text_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"562cce8b27dea629"}},{"code_sha256_prefix":"bb35e3ac741bb2c9","entry":"get_mm_adapter_state_maybe_zero_3","repo":"rayruibochen/self-filter","repo_kind":"official","path":"self_filter/self_filter_trainer.py","file_url":"https://github.com/rayruibochen/self-filter/blob/HEAD/self_filter/self_filter_trainer.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"bb35e3ac741bb2c9"}},{"code_sha256_prefix":"fa1225dfac92bc0d","entry":"get_peft_state_maybe_zero_3","repo":"rayruibochen/self-filter","repo_kind":"official","path":"self_filter/stage1.py","file_url":"https://github.com/rayruibochen/self-filter/blob/HEAD/self_filter/stage1.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"fa1225dfac92bc0d"}},{"code_sha256_prefix":"1c53657305b66e9f","entry":"get_peft_state_non_lora_maybe_zero_3","repo":"rayruibochen/self-filter","repo_kind":"official","path":"self_filter/stage1.py","file_url":"https://github.com/rayruibochen/self-filter/blob/HEAD/self_filter/stage1.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"1c53657305b66e9f"}},{"code_sha256_prefix":"735025744c1ab0cf","entry":"maybe_zero_3","repo":"rayruibochen/self-filter","repo_kind":"official","path":"self_filter/self_filter_trainer.py","file_url":"https://github.com/rayruibochen/self-filter/blob/HEAD/self_filter/self_filter_trainer.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"735025744c1ab0cf"}},{"code_sha256_prefix":"616ffbdc154ed2d8","entry":"maybe_zero_3","repo":"rayruibochen/self-filter","repo_kind":"official","path":"self_filter/stage1.py","file_url":"https://github.com/rayruibochen/self-filter/blob/HEAD/self_filter/stage1.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"616ffbdc154ed2d8"}},{"code_sha256_prefix":"10d15d374cd997c3","entry":"produce_scores_difficulty","repo":"rayruibochen/self-filter","repo_kind":"official","path":"self_filter/stage2.py","file_url":"https://github.com/rayruibochen/self-filter/blob/HEAD/self_filter/stage2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"10d15d374cd997c3"}},{"code_sha256_prefix":"841b22f7ec0e92b2","entry":"query_gpt","repo":"rayruibochen/self-filter","repo_kind":"official","path":"data_process/query_gpt_text_models.py","file_url":"https://github.com/rayruibochen/self-filter/blob/HEAD/data_process/query_gpt_text_models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"841b22f7ec0e92b2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}