{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/meteor-mamba-based-traversal-of-rationale-for","title":"Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models","arxiv_id":"2405.15574","date":"2024-05-24","proceeding":null,"authors":["Byung-Kwan Lee","Chae Won Kim","Beomchan Park","Yong Man Ro"],"abstract":"The rapid development of large language and vision models (LLVMs) has been driven by advances in visual instruction tuning. Recently, open-source LLVMs have curated high-quality visual instruction tuning datasets and utilized additional vision encoders or multiple computer vision models in order to narrow the performance gap with powerful closed-source LLVMs. These advancements are attributed to multifaceted information required for diverse capabilities, including fundamental image understanding, real-world knowledge about common-sense and non-object concepts (e.g., charts, diagrams, symbols, signs, and math problems), and step-by-step procedures for solving complex questions. Drawing from the multifaceted information, we present a new efficient LLVM, Mamba-based traversal of rationales (Meteor), which leverages multifaceted rationale to enhance understanding and answering capabilities. To embed lengthy rationales containing abundant information, we employ the Mamba architecture, capable of processing sequential data with linear time complexity. We introduce a new concept of traversal of rationale that facilitates efficient embedding of rationale. Subsequently, the backbone multimodal language model (MLM) is trained to generate answers with the aid of rationale. Through these steps, Meteor achieves significant improvements in vision language performances across multiple evaluation benchmarks requiring diverse capabilities, without scaling up the model size or employing additional vision encoders and computer vision models.","url_abs":"https://arxiv.org/abs/2405.15574v4","url_pdf":"https://arxiv.org/pdf/2405.15574v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"meteor-mamba-based-traversal-of-rationale-for","repo_url":"https://github.com/byungkwanlee/meteor","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"mamba","task_name":"Mamba"},{"task_slug":"math","task_name":"Math"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-mm-vet","task":"Visual Question Answering","dataset":"MM-Vet","model":"Meteor","rank_in_archive_order":45,"of":231,"metrics":{"GPT-4 score":"57.3","Params":"7B"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.15574","atlas_url":"https://app.syntology.ai/?focus=2405.15574","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.15574"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/byungkwanlee/meteor","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":8,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ff5aa85ec44db5ee","entry":"normalize_extracted_answer","repo":"byungkwanlee/meteor","repo_kind":"official","path":"eval/mathvista/calculate_score.py","file_url":"https://github.com/byungkwanlee/meteor/blob/HEAD/eval/mathvista/calculate_score.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff5aa85ec44db5ee"}},{"code_sha256_prefix":"c0aae1f053da2578","entry":"repeat_kv","repo":"byungkwanlee/meteor","repo_kind":"official","path":"meteor/arch/modeling_internlm2.py","file_url":"https://github.com/byungkwanlee/meteor/blob/HEAD/meteor/arch/modeling_internlm2.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0aae1f053da2578"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"byungkwanlee/meteor","repo_kind":"official","path":"meteor/arch/modeling_internlm2.py","file_url":"https://github.com/byungkwanlee/meteor/blob/HEAD/meteor/arch/modeling_internlm2.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"051ebcfa8673a02e","entry":"apply_rotary_pos_emb","repo":"byungkwanlee/meteor","repo_kind":"official","path":"meteor/arch/modeling_internlm2.py","file_url":"https://github.com/byungkwanlee/meteor/blob/HEAD/meteor/arch/modeling_internlm2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"051ebcfa8673a02e"}},{"code_sha256_prefix":"c28eb86a8d14e4ed","entry":"get_most_similar","repo":"byungkwanlee/meteor","repo_kind":"official","path":"eval/mathvista/calculate_score.py","file_url":"https://github.com/byungkwanlee/meteor/blob/HEAD/eval/mathvista/calculate_score.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c28eb86a8d14e4ed"}},{"code_sha256_prefix":"067e5c232728f6ca","entry":"load_meteor","repo":"byungkwanlee/meteor","repo_kind":"official","path":"meteor/load_meteor.py","file_url":"https://github.com/byungkwanlee/meteor/blob/HEAD/meteor/load_meteor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"067e5c232728f6ca"}},{"code_sha256_prefix":"3e0610ca85698e47","entry":"load_mmamba","repo":"byungkwanlee/meteor","repo_kind":"official","path":"meteor/load_mmamba.py","file_url":"https://github.com/byungkwanlee/meteor/blob/HEAD/meteor/load_mmamba.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e0610ca85698e47"}},{"code_sha256_prefix":"e7a23ea3695f6437","entry":"safe_equal","repo":"byungkwanlee/meteor","repo_kind":"official","path":"eval/mathvista/calculate_score.py","file_url":"https://github.com/byungkwanlee/meteor/blob/HEAD/eval/mathvista/calculate_score.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7a23ea3695f6437"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}