{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sq-llava-self-questioning-for-large-vision","title":"SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant","arxiv_id":"2403.11299","date":"2024-03-17","proceeding":null,"authors":["Guohao Sun","Can Qin","Jiamian Wang","Zeyuan Chen","ran Xu","Zhiqiang Tao"],"abstract":"Recent advances in vision-language models have shown notable generalization in broad tasks through visual instruction tuning. However, bridging the gap between the pre-trained vision encoder and the large language models (LLMs) becomes the whole network's bottleneck. To improve cross-modality alignment, existing works usually consider more visual instruction data covering a broader range of vision tasks to fine-tune the model for question-answering, which, however, is costly to obtain and has not thoroughly explored the rich contextual information contained in images. This paper first attempts to harness the overlooked context within visual instruction data, training the model to self-supervised \"learning\" how to ask high-quality questions. In this way, we introduce a novel framework named SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant. SQ-LLaVA exhibits proficiency in generating flexible and meaningful image-related questions while analyzing the visual clue and prior language knowledge, signifying an advanced level of generalized visual understanding. Moreover, fine-tuning SQ-LLaVA on higher-quality instruction data shows a performance improvement compared with traditional visual-instruction tuning methods. This improvement highlights the efficacy of self-questioning techniques in achieving a deeper and more nuanced comprehension of visual content across various contexts.","url_abs":"https://arxiv.org/abs/2403.11299v2","url_pdf":"https://arxiv.org/pdf/2403.11299v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sq-llava-self-questioning-for-large-vision","repo_url":"https://github.com/heliossun/sq-llava","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"sq-llava-self-questioning-for-large-vision","repo_url":"https://github.com/heliossun/stllava-med","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-mm-vet","task":"Visual Question Answering","dataset":"MM-Vet","model":"SQ-LLaVA∗","rank_in_archive_order":117,"of":231,"metrics":{"GPT-4 score":"39.7"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-mm-vet","task":"Visual Question Answering","dataset":"MM-Vet","model":"SQ-LLaVA","rank_in_archive_order":157,"of":231,"metrics":{"GPT-4 score":"35.5"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2403.11299","atlas_url":"https://app.syntology.ai/?focus=2403.11299","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.11299"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/heliossun/sq-llava","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/heliossun/stllava-med","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/heliossun/SQ-LLaVA","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"ran_honours":1,"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d02a03fedb337e4b","entry":"BroadMultiHeadAttention","repo":"heliossun/SQ-LLaVA","repo_kind":"official","path":"sqllava/model/crossattentionLayer.py","file_url":"https://github.com/heliossun/SQ-LLaVA/blob/HEAD/sqllava/model/crossattentionLayer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d02a03fedb337e4b"}},{"code_sha256_prefix":"437f9ed9e855f0de","entry":"CrossAttentionLayer","repo":"heliossun/SQ-LLaVA","repo_kind":"official","path":"sqllava/model/crossattentionLayer.py","file_url":"https://github.com/heliossun/SQ-LLaVA/blob/HEAD/sqllava/model/crossattentionLayer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"437f9ed9e855f0de"}},{"code_sha256_prefix":"9b3c1cb391672ccb","entry":"load_image","repo":"heliossun/sq-llava","repo_kind":"official","path":"sqllava/eval/run_llava.py","file_url":"https://github.com/heliossun/sq-llava/blob/HEAD/sqllava/eval/run_llava.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b3c1cb391672ccb"}},{"code_sha256_prefix":"10893c4608c08075","entry":"split_to_even_chunks","repo":"heliossun/sq-llava","repo_kind":"official","path":"llava_trainer.py","file_url":"https://github.com/heliossun/sq-llava/blob/HEAD/llava_trainer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10893c4608c08075"}},{"code_sha256_prefix":"bb35e3ac741bb2c9","entry":"get_mm_adapter_state_maybe_zero_3","repo":"heliossun/sq-llava","repo_kind":"official","path":"llava_trainer.py","file_url":"https://github.com/heliossun/sq-llava/blob/HEAD/llava_trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb35e3ac741bb2c9"}},{"code_sha256_prefix":"735025744c1ab0cf","entry":"maybe_zero_3","repo":"heliossun/sq-llava","repo_kind":"official","path":"llava_trainer.py","file_url":"https://github.com/heliossun/sq-llava/blob/HEAD/llava_trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"735025744c1ab0cf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}