{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-robust-multi-modal-reasoning-via","title":"Towards Robust Multi-Modal Reasoning via Model Selection","arxiv_id":"2310.08446","date":"2023-10-12","proceeding":null,"authors":["Xiangyan Liu","Rongxue Li","Wei Ji","Tao Lin"],"abstract":"The reasoning capabilities of LLM (Large Language Model) are widely acknowledged in recent research, inspiring studies on tool learning and autonomous agents. LLM serves as the \"brain\" of the agent, orchestrating multiple tools for collaborative multi-step task solving. Unlike methods invoking tools like calculators or weather APIs for straightforward tasks, multi-modal agents excel by integrating diverse AI models for complex challenges. However, current multi-modal agents neglect the significance of model selection: they primarily focus on the planning and execution phases, and will only invoke predefined task-specific models for each subtask, making the execution fragile. Meanwhile, other traditional model selection methods are either incompatible with or suboptimal for the multi-modal agent scenarios, due to ignorance of dependencies among subtasks arising by multi-step reasoning. To this end, we identify the key challenges therein and propose the $\\textit{M}^3$ framework as a plug-in with negligible runtime overhead at test-time. This framework improves model selection and bolsters the robustness of multi-modal agents in multi-step reasoning. In the absence of suitable benchmarks, we create MS-GQA, a new dataset specifically designed to investigate the model selection challenge in multi-modal agents. Our experiments reveal that our framework enables dynamic model selection, considering both user inputs and subtask dependencies, thereby robustifying the overall reasoning process. Our code and benchmark: https://github.com/LINs-lab/M3.","url_abs":"https://arxiv.org/abs/2310.08446v2","url_pdf":"https://arxiv.org/pdf/2310.08446v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-robust-multi-modal-reasoning-via","repo_url":"https://github.com/LINs-lab/M3","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"model-selection","task_name":"Model Selection"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2310.08446","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.08446"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/LINs-lab/M3","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"unverified":3},"by_repo_kind":{"official":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f93b1f963e736c8b","entry":"create_graph_from_program","repo":"LINs-lab/M3","repo_kind":"official","path":"MS-GQA/code/preprocess.py","file_url":"https://github.com/LINs-lab/M3/blob/HEAD/MS-GQA/code/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f93b1f963e736c8b"}},{"code_sha256_prefix":"54f795d322b7185d","entry":"get_ori_task_types","repo":"LINs-lab/M3","repo_kind":"official","path":"MS-GQA/code/run_metagl.py","file_url":"https://github.com/LINs-lab/M3/blob/HEAD/MS-GQA/code/run_metagl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"54f795d322b7185d"}},{"code_sha256_prefix":"6789f8911c900d76","entry":"parse_step","repo":"LINs-lab/M3","repo_kind":"official","path":"MS-GQA/code/preprocess.py","file_url":"https://github.com/LINs-lab/M3/blob/HEAD/MS-GQA/code/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6789f8911c900d76"}},{"code_sha256_prefix":"d8d9625bb4bcbddf","entry":"build_dataset","repo":"LINs-lab/M3","repo_kind":"official","path":"MS-GQA/code/run_metagl.py","file_url":"https://github.com/LINs-lab/M3/blob/HEAD/MS-GQA/code/run_metagl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d8d9625bb4bcbddf"}},{"code_sha256_prefix":"a1cb3bb3f6e48673","entry":"evaluate_model","repo":"LINs-lab/M3","repo_kind":"official","path":"MS-GQA/code/run_m3.py","file_url":"https://github.com/LINs-lab/M3/blob/HEAD/MS-GQA/code/run_m3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a1cb3bb3f6e48673"}},{"code_sha256_prefix":"80cf4f2a02aa0335","entry":"evaluate_model","repo":"LINs-lab/M3","repo_kind":"official","path":"MS-GQA/code/run_ncf++.py","file_url":"https://github.com/LINs-lab/M3/blob/HEAD/MS-GQA/code/run_ncf%2B%2B.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"80cf4f2a02aa0335"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}