{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-efficient-language-and-vision-assistants","title":"On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning","arxiv_id":"2406.11823","date":"2024-06-17","proceeding":null,"authors":["Geewook Kim","Minjoon Seo"],"abstract":"Recent advancements in language and vision assistants have showcased impressive capabilities but suffer from a lack of transparency, limiting broader research and reproducibility. While open-source models handle general image tasks effectively, they face challenges with the high computational demands of complex visually-situated text understanding. Such tasks often require increased token inputs and large vision modules to harness high-resolution information. Striking a balance between model size and data importance remains an open question. This study aims to redefine the design of vision-language models by identifying key components and creating efficient models with constrained inference costs. By strategically formulating datasets, optimizing vision modules, and enhancing supervision techniques, we achieve significant improvements in inference throughput while maintaining high performance. Extensive experiments across models ranging from 160M to 13B parameters offer insights into model optimization. We will fully open-source our codebase, models, and datasets at https://github.com/naver-ai/elva.","url_abs":"https://arxiv.org/abs/2406.11823v2","url_pdf":"https://arxiv.org/pdf/2406.11823v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-efficient-language-and-vision-assistants","repo_url":"https://github.com/naver-ai/elva","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"document-ai","task_name":"Document AI"},{"task_slug":"model-optimization","task_name":"Model Optimization"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.11823","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.11823"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/naver-ai/elva","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":4,"ran_violates":1,"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"7e03b180fa317c9a","entry":"divide_to_patches","repo":"naver-ai/elva","repo_kind":"official","path":"Elva/mm_utils.py","file_url":"https://github.com/naver-ai/elva/blob/HEAD/Elva/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e03b180fa317c9a"}},{"code_sha256_prefix":"8a203abf91abfcad","entry":"get_chunk","repo":"naver-ai/elva","repo_kind":"official","path":"Elva/model_vqa_parsing.py","file_url":"https://github.com/naver-ai/elva/blob/HEAD/Elva/model_vqa_parsing.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8a203abf91abfcad"}},{"code_sha256_prefix":"763f6fe20bd52fbb","entry":"parse_score","repo":"naver-ai/elva","repo_kind":"official","path":"Elva/eval_gpt_review_parsing_bench.py","file_url":"https://github.com/naver-ai/elva/blob/HEAD/Elva/eval_gpt_review_parsing_bench.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"763f6fe20bd52fbb"}},{"code_sha256_prefix":"ef6b9820c5e60b06","entry":"resize_and_pad_image","repo":"naver-ai/elva","repo_kind":"official","path":"Elva/mm_utils.py","file_url":"https://github.com/naver-ai/elva/blob/HEAD/Elva/mm_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ef6b9820c5e60b06"}},{"code_sha256_prefix":"3999ff487573f32c","entry":"select_best_resolution","repo":"naver-ai/elva","repo_kind":"official","path":"Elva/mm_utils.py","file_url":"https://github.com/naver-ai/elva/blob/HEAD/Elva/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3999ff487573f32c"}},{"code_sha256_prefix":"abe4f5d245fa207a","entry":"split_list","repo":"naver-ai/elva","repo_kind":"official","path":"Elva/model_vqa_parsing.py","file_url":"https://github.com/naver-ai/elva/blob/HEAD/Elva/model_vqa_parsing.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"abe4f5d245fa207a"}},{"code_sha256_prefix":"d2f4c2f963c75be2","entry":"build_vision_tower","repo":"naver-ai/elva","repo_kind":"official","path":"Elva/encoder_builder.py","file_url":"https://github.com/naver-ai/elva/blob/HEAD/Elva/encoder_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d2f4c2f963c75be2"}},{"code_sha256_prefix":"a5b1f40fc357a806","entry":"get_eval","repo":"naver-ai/elva","repo_kind":"official","path":"Elva/eval_gpt_review_parsing_bench.py","file_url":"https://github.com/naver-ai/elva/blob/HEAD/Elva/eval_gpt_review_parsing_bench.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a5b1f40fc357a806"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}