{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/medagentboard-benchmarking-multi-agent","title":"MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks","arxiv_id":"2505.12371","date":"2025-05-18","proceeding":null,"authors":["Yinghao Zhu","Ziyi He","Haoran Hu","Xiaochen Zheng","Xichen Zhang","Zixiang Wang","Junyi Gao","Liantao Ma","Lequan Yu"],"abstract":"The rapid advancement of Large Language Models (LLMs) has stimulated interest in multi-agent collaboration for addressing complex medical tasks. However, the practical advantages of multi-agent collaboration approaches remain insufficiently understood. Existing evaluations often lack generalizability, failing to cover diverse tasks reflective of real-world clinical practice, and frequently omit rigorous comparisons against both single-LLM-based and established conventional methods. To address this critical gap, we introduce MedAgentBoard, a comprehensive benchmark for the systematic evaluation of multi-agent collaboration, single-LLM, and conventional approaches. MedAgentBoard encompasses four diverse medical task categories: (1) medical (visual) question answering, (2) lay summary generation, (3) structured Electronic Health Record (EHR) predictive modeling, and (4) clinical workflow automation, across text, medical images, and structured EHR data. Our extensive experiments reveal a nuanced landscape: while multi-agent collaboration demonstrates benefits in specific scenarios, such as enhancing task completeness in clinical workflow automation, it does not consistently outperform advanced single LLMs (e.g., in textual medical QA) or, critically, specialized conventional methods that generally maintain better performance in tasks like medical VQA and EHR-based prediction. MedAgentBoard offers a vital resource and actionable insights, emphasizing the necessity of a task-specific, evidence-based approach to selecting and developing AI solutions in medicine. It underscores that the inherent complexity and overhead of multi-agent collaboration must be carefully weighed against tangible performance gains. All code, datasets, detailed prompts, and experimental results are open-sourced at https://medagentboard.netlify.app/.","url_abs":"https://arxiv.org/abs/2505.12371v1","url_pdf":"https://arxiv.org/pdf/2505.12371v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"medagentboard-benchmarking-multi-agent","repo_url":"https://github.com/yhzhu99/medagentboard","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"medical-visual-question-answering","task_name":"Medical Visual Question Answering"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.12371","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.12371"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/HAIRLAB/Pre_Surv_COVID_19","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/attal-kush/PLABA","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/huggingface/smolagents","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yhzhu99/medagentboard","reach":{"status":"ok"}}],"summary":{"ran":1,"unverified":15},"by_repo_kind":{"found_in_text":{"samples":16,"ran":1,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cfeb7cd7a490c27c","entry":"parse_json_if_needed","repo":"huggingface/smolagents","repo_kind":"found_in_text","path":"src/smolagents/models.py","file_url":"https://github.com/huggingface/smolagents/blob/HEAD/src/smolagents/models.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cfeb7cd7a490c27c"}},{"code_sha256_prefix":"90e6153dd6663fa7","entry":"StratifiedKFold_func","repo":"HAIRLAB/Pre_Surv_COVID_19","repo_kind":"found_in_text","path":"utils_features_selection.py","file_url":"https://github.com/HAIRLAB/Pre_Surv_COVID_19/blob/HEAD/utils_features_selection.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90e6153dd6663fa7"}},{"code_sha256_prefix":"e9feaed780d7263b","entry":"StratifiedKFold_func_with_features_sel","repo":"HAIRLAB/Pre_Surv_COVID_19","repo_kind":"found_in_text","path":"utils_features_selection.py","file_url":"https://github.com/HAIRLAB/Pre_Surv_COVID_19/blob/HEAD/utils_features_selection.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e9feaed780d7263b"}},{"code_sha256_prefix":"990f6625f3b963da","entry":"get_dict_from_nested_dataclasses","repo":"huggingface/smolagents","repo_kind":"found_in_text","path":"src/smolagents/models.py","file_url":"https://github.com/huggingface/smolagents/blob/HEAD/src/smolagents/models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"990f6625f3b963da"}},{"code_sha256_prefix":"058f2cecb716999e","entry":"get_imports","repo":"huggingface/smolagents","repo_kind":"found_in_text","path":"src/smolagents/_function_type_hints_utils.py","file_url":"https://github.com/huggingface/smolagents/blob/HEAD/src/smolagents/_function_type_hints_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"058f2cecb716999e"}},{"code_sha256_prefix":"ad273f108393d5cf","entry":"get_json_schema","repo":"huggingface/smolagents","repo_kind":"found_in_text","path":"src/smolagents/_function_type_hints_utils.py","file_url":"https://github.com/huggingface/smolagents/blob/HEAD/src/smolagents/_function_type_hints_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ad273f108393d5cf"}},{"code_sha256_prefix":"71f9eaf9b1c3790a","entry":"get_package_name","repo":"huggingface/smolagents","repo_kind":"found_in_text","path":"src/smolagents/_function_type_hints_utils.py","file_url":"https://github.com/huggingface/smolagents/blob/HEAD/src/smolagents/_function_type_hints_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"71f9eaf9b1c3790a"}},{"code_sha256_prefix":"608d577aef4c28b4","entry":"handle_agent_output_types","repo":"huggingface/smolagents","repo_kind":"found_in_text","path":"src/smolagents/agent_types.py","file_url":"https://github.com/huggingface/smolagents/blob/HEAD/src/smolagents/agent_types.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"608d577aef4c28b4"}},{"code_sha256_prefix":"f97aabe235dec048","entry":"is_number","repo":"HAIRLAB/Pre_Surv_COVID_19","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/HAIRLAB/Pre_Surv_COVID_19/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f97aabe235dec048"}},{"code_sha256_prefix":"d8b6294050eecce3","entry":"merge_data_by_sliding_window","repo":"HAIRLAB/Pre_Surv_COVID_19","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/HAIRLAB/Pre_Surv_COVID_19/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d8b6294050eecce3"}},{"code_sha256_prefix":"deb5a11caf7f2cf0","entry":"nodunder_getattr","repo":"huggingface/smolagents","repo_kind":"found_in_text","path":"src/smolagents/local_python_executor.py","file_url":"https://github.com/huggingface/smolagents/blob/HEAD/src/smolagents/local_python_executor.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"deb5a11caf7f2cf0"}},{"code_sha256_prefix":"a5e997d9057b1af6","entry":"populate_template","repo":"huggingface/smolagents","repo_kind":"found_in_text","path":"src/smolagents/agents.py","file_url":"https://github.com/huggingface/smolagents/blob/HEAD/src/smolagents/agents.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a5e997d9057b1af6"}},{"code_sha256_prefix":"bd1a6c1715adc6c4","entry":"read","repo":"HAIRLAB/Pre_Surv_COVID_19","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/HAIRLAB/Pre_Surv_COVID_19/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd1a6c1715adc6c4"}},{"code_sha256_prefix":"eac7bc569b8add8a","entry":"read_train_data","repo":"HAIRLAB/Pre_Surv_COVID_19","repo_kind":"found_in_text","path":"utils_features_selection.py","file_url":"https://github.com/HAIRLAB/Pre_Surv_COVID_19/blob/HEAD/utils_features_selection.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eac7bc569b8add8a"}},{"code_sha256_prefix":"36443b04f5596ace","entry":"remove_content_after_stop_sequences","repo":"huggingface/smolagents","repo_kind":"found_in_text","path":"src/smolagents/models.py","file_url":"https://github.com/huggingface/smolagents/blob/HEAD/src/smolagents/models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"36443b04f5596ace"}},{"code_sha256_prefix":"b51a3753b2d3d810","entry":"safer_eval","repo":"huggingface/smolagents","repo_kind":"found_in_text","path":"src/smolagents/local_python_executor.py","file_url":"https://github.com/huggingface/smolagents/blob/HEAD/src/smolagents/local_python_executor.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b51a3753b2d3d810"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}