{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/phantom-of-latent-for-large-language-and","title":"Phantom of Latent for Large Language and Vision Models","arxiv_id":"2409.14713","date":"2024-09-23","proceeding":null,"authors":["Byung-Kwan Lee","Sangyun Chung","Chae Won Kim","Beomchan Park","Yong Man Ro"],"abstract":"The success of visual instruction tuning has accelerated the development of large language and vision models (LLVMs). Following the scaling laws of instruction-tuned large language models (LLMs), LLVMs either have further increased their sizes, reaching 26B, 34B, and even 80B parameters. While this increase in model size has yielded significant performance gains, it demands substantially more hardware resources for both training and inference. Consequently, there naturally exists a strong need for efficient LLVMs that achieve the performance of larger models while being smaller in size. To achieve this need, we present a new efficient LLVM family with model sizes of 0.5B, 1.8B, 3.8B, and 7B parameters, Phantom, which significantly enhances learning capabilities within limited structures. By temporarily increasing the latent hidden dimension during multi-head self-attention (MHSA), we make LLVMs prepare to look and understand much more vision-language knowledge on the latent, without substantially increasing physical model sizes. To maximize its advantage, we introduce Phantom Optimization (PO) using both autoregressive supervised fine-tuning (SFT) and direct preference optimization (DPO)-like concept, which effectively follows correct answers while eliminating incorrect and ambiguous ones. Phantom outperforms numerous larger open- and closed-source LLVMs, positioning itself as a leading solution in the landscape of efficient LLVMs.","url_abs":"https://arxiv.org/abs/2409.14713v1","url_pdf":"https://arxiv.org/pdf/2409.14713v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"phantom-of-latent-for-large-language-and","repo_url":"https://github.com/byungkwanlee/phantom","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-mm-vet","task":"Visual Question Answering","dataset":"MM-Vet","model":"Phantom-7B","rank_in_archive_order":11,"of":231,"metrics":{"GPT-4 score":"70.8"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.14713","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.14713"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/byungkwanlee/phantom","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":3},"by_repo_kind":{"official":{"samples":4,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8b901d87160993c2","entry":"normalize_extracted_answer","repo":"byungkwanlee/phantom","repo_kind":"official","path":"eval/mathvista/calculate_score.py","file_url":"https://github.com/byungkwanlee/phantom/blob/HEAD/eval/mathvista/calculate_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b901d87160993c2"}},{"code_sha256_prefix":"c28eb86a8d14e4ed","entry":"get_most_similar","repo":"byungkwanlee/phantom","repo_kind":"official","path":"eval/mathvista/calculate_score.py","file_url":"https://github.com/byungkwanlee/phantom/blob/HEAD/eval/mathvista/calculate_score.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c28eb86a8d14e4ed"}},{"code_sha256_prefix":"db11a804a6adefad","entry":"load_model","repo":"byungkwanlee/phantom","repo_kind":"official","path":"model/load_model.py","file_url":"https://github.com/byungkwanlee/phantom/blob/HEAD/model/load_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db11a804a6adefad"}},{"code_sha256_prefix":"e7a23ea3695f6437","entry":"safe_equal","repo":"byungkwanlee/phantom","repo_kind":"official","path":"eval/mathvista/calculate_score.py","file_url":"https://github.com/byungkwanlee/phantom/blob/HEAD/eval/mathvista/calculate_score.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7a23ea3695f6437"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}