{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/llava-more-a-comparative-study-of-llms-and","title":"LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning","arxiv_id":"2503.15621","date":"2025-03-19","proceeding":null,"authors":["Federico Cocchi","Nicholas Moratelli","Davide Caffagni","Sara Sarto","Lorenzo Baraldi","Marcella Cornia","Rita Cucchiara"],"abstract":"Recent progress in Multimodal Large Language Models (MLLMs) has highlighted the critical roles of both the visual backbone and the underlying language model. While prior work has primarily focused on scaling these components to billions of parameters, the trade-offs between model size, architecture, and performance remain underexplored. Additionally, inconsistencies in training data and evaluation protocols have hindered direct comparisons, making it difficult to derive optimal design choices. In this paper, we introduce LLaVA-MORE, a new family of MLLMs that integrates recent language models with diverse visual backbones. To ensure fair comparisons, we employ a unified training protocol applied consistently across all architectures. Our analysis systematically explores both small- and medium-scale LLMs -- including Phi-4, LLaMA-3.1, and Gemma-2 -- to evaluate multimodal reasoning, generation, and instruction following, while examining the relationship between model size and performance. Beyond evaluating the LLM impact on final results, we conduct a comprehensive study of various visual encoders, ranging from CLIP-based architectures to alternatives such as DINOv2, SigLIP, and SigLIP2. Additional experiments investigate the effects of increased image resolution and variations in pre-training datasets. Overall, our results provide insights into the design of more effective MLLMs, offering a reproducible evaluation framework that facilitates direct comparisons and can guide future model development. Our source code and trained models are publicly available at: https://github.com/aimagelab/LLaVA-MORE.","url_abs":"https://arxiv.org/abs/2503.15621v1","url_pdf":"https://arxiv.org/pdf/2503.15621v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"llava-more-a-comparative-study-of-llms-and","repo_url":"https://github.com/aimagelab/LLaVA-MORE","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"multimodal-reasoning","task_name":"Multimodal Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2503.15621","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.15621"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aimagelab/LLaVA-MORE","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"ran":3,"ran_violates":2,"ran_fixture":2,"unverified":6},"by_repo_kind":{"official":{"samples":13,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"94fda0761ce0d5b9","entry":"convert_caps","repo":"aimagelab/LLaVA-MORE","repo_kind":"official","path":"src/llava/eval/eval_science_qa.py","file_url":"https://github.com/aimagelab/LLaVA-MORE/blob/HEAD/src/llava/eval/eval_science_qa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"94fda0761ce0d5b9"}},{"code_sha256_prefix":"7e03b180fa317c9a","entry":"divide_to_patches","repo":"aimagelab/LLaVA-MORE","repo_kind":"official","path":"src/llava/mm_utils.py","file_url":"https://github.com/aimagelab/LLaVA-MORE/blob/HEAD/src/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e03b180fa317c9a"}},{"code_sha256_prefix":"fb48214e8ea84c17","entry":"get_eval","repo":"aimagelab/LLaVA-MORE","repo_kind":"official","path":"src/llava/eval/eval_gpt_review_bench.py","file_url":"https://github.com/aimagelab/LLaVA-MORE/blob/HEAD/src/llava/eval/eval_gpt_review_bench.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb48214e8ea84c17"}},{"code_sha256_prefix":"36e875f85baa4565","entry":"get_pred_idx","repo":"aimagelab/LLaVA-MORE","repo_kind":"official","path":"src/llava/eval/eval_science_qa.py","file_url":"https://github.com/aimagelab/LLaVA-MORE/blob/HEAD/src/llava/eval/eval_science_qa.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"36e875f85baa4565"}},{"code_sha256_prefix":"763f6fe20bd52fbb","entry":"parse_score","repo":"aimagelab/LLaVA-MORE","repo_kind":"official","path":"src/llava/eval/eval_gpt_review.py","file_url":"https://github.com/aimagelab/LLaVA-MORE/blob/HEAD/src/llava/eval/eval_gpt_review.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"763f6fe20bd52fbb"}},{"code_sha256_prefix":"468eedeba67f1b00","entry":"resize_and_pad_image","repo":"aimagelab/LLaVA-MORE","repo_kind":"official","path":"src/llava/mm_utils.py","file_url":"https://github.com/aimagelab/LLaVA-MORE/blob/HEAD/src/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"468eedeba67f1b00"}},{"code_sha256_prefix":"3999ff487573f32c","entry":"select_best_resolution","repo":"aimagelab/LLaVA-MORE","repo_kind":"official","path":"src/llava/mm_utils.py","file_url":"https://github.com/aimagelab/LLaVA-MORE/blob/HEAD/src/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3999ff487573f32c"}},{"code_sha256_prefix":"55c32993da87759b","entry":"unpad_image","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"55c32993da87759b"}},{"code_sha256_prefix":"64a3ae0bc1b51280","entry":"get_eval","repo":"aimagelab/LLaVA-MORE","repo_kind":"official","path":"src/llava/eval/eval_gpt_review.py","file_url":"https://github.com/aimagelab/LLaVA-MORE/blob/HEAD/src/llava/eval/eval_gpt_review.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"64a3ae0bc1b51280"}},{"code_sha256_prefix":"d76907fa6a0240e6","entry":"get_logger","repo":"aimagelab/LLaVA-MORE","repo_kind":"official","path":"src/llava/utils.py","file_url":"https://github.com/aimagelab/LLaVA-MORE/blob/HEAD/src/llava/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d76907fa6a0240e6"}},{"code_sha256_prefix":"ab7f4164814e1584","entry":"get_model","repo":"aimagelab/LLaVA-MORE","repo_kind":"official","path":"src/llava/load_utils.py","file_url":"https://github.com/aimagelab/LLaVA-MORE/blob/HEAD/src/llava/load_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ab7f4164814e1584"}},{"code_sha256_prefix":"66a05b5db47523b9","entry":"get_pred_idx","repo":"aimagelab/LLaVA-MORE","repo_kind":"official","path":"src/llava/eval/eval_science_qa_gpt4.py","file_url":"https://github.com/aimagelab/LLaVA-MORE/blob/HEAD/src/llava/eval/eval_science_qa_gpt4.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66a05b5db47523b9"}},{"code_sha256_prefix":"8c3e17cd14e0016b","entry":"get_tokenizer","repo":"aimagelab/LLaVA-MORE","repo_kind":"official","path":"src/llava/load_utils.py","file_url":"https://github.com/aimagelab/LLaVA-MORE/blob/HEAD/src/llava/load_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c3e17cd14e0016b"}},{"code_sha256_prefix":"f9939a84b9a65279","entry":"violates_moderation","repo":"aimagelab/LLaVA-MORE","repo_kind":"official","path":"src/llava/utils.py","file_url":"https://github.com/aimagelab/LLaVA-MORE/blob/HEAD/src/llava/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9939a84b9a65279"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}