{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2603-14326","title":"ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation","arxiv_id":"2603.14326","date":"2026-03-15","proceeding":null,"authors":["Jungwoo Oh","Hyunseung Chung","Junhee Lee","Min-Gyu Kim","Hangyul Yoon","Ki Seong Lee","Youngchae Lee","Muhan Yeo","Edward Choi"],"abstract":"While Multimodal Large Language Models (MLLMs) show promising performance in automated electrocardiogram interpretation, it remains unclear whether they genuinely perform actual step-by-step reasoning or just rely on superficial visual cues. To investigate this, we introduce \\textbf{ECG-Reasoning-Benchmark}, a novel multi-turn evaluation framework comprising over 6,400 samples to systematically assess step-by-step reasoning across 17 core ECG diagnoses. Our comprehensive evaluation of state-of-the-art models reveals a critical failure in executing multi-step logical deduction. Although models possess the medical knowledge to retrieve clinical criteria for a diagnosis, they exhibit near-zero success rates (6% Completion) in maintaining a complete reasoning chain, primarily failing to ground the corresponding ECG findings to the actual visual evidence in the ECG signal. These results demonstrate that current MLLMs bypass actual visual interpretation, exposing a critical flaw in existing training paradigms and underscoring the necessity for robust, reasoning-centric medical AI. The code and data are available at https://github.com/Jwoo5/ecg-reasoning-benchmark.","url_abs":"https://arxiv.org/abs/2603.14326","url_pdf":"https://arxiv.org/pdf/2603.14326","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2603.14326","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2603.14326"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Jwoo5/ecg-reasoning-benchmark","reach":null}],"summary":{"unverified":5},"by_repo_kind":{"found_in_text":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8e43b05662c26657","entry":"base64_image_encoder","repo":"Jwoo5/ecg-reasoning-benchmark","repo_kind":"found_in_text","path":"ecg_reasoning_benchmark/utils.py","file_url":"https://github.com/Jwoo5/ecg-reasoning-benchmark/blob/HEAD/ecg_reasoning_benchmark/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e43b05662c26657"}},{"code_sha256_prefix":"965ab11a62addd96","entry":"evaluate_results","repo":"Jwoo5/ecg-reasoning-benchmark","repo_kind":"found_in_text","path":"ecg_reasoning_benchmark/evaluation.py","file_url":"https://github.com/Jwoo5/ecg-reasoning-benchmark/blob/HEAD/ecg_reasoning_benchmark/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"965ab11a62addd96"}},{"code_sha256_prefix":"288aecfb4465e958","entry":"is_retryable_error","repo":"Jwoo5/ecg-reasoning-benchmark","repo_kind":"found_in_text","path":"ecg_reasoning_benchmark/evaluators/gemini.py","file_url":"https://github.com/Jwoo5/ecg-reasoning-benchmark/blob/HEAD/ecg_reasoning_benchmark/evaluators/gemini.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"288aecfb4465e958"}},{"code_sha256_prefix":"079913ef2f262e91","entry":"load_data","repo":"Jwoo5/ecg-reasoning-benchmark","repo_kind":"found_in_text","path":"ecg_reasoning_benchmark/inference.py","file_url":"https://github.com/Jwoo5/ecg-reasoning-benchmark/blob/HEAD/ecg_reasoning_benchmark/inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"079913ef2f262e91"}},{"code_sha256_prefix":"e186589d35e07f11","entry":"make_letter_indexed","repo":"Jwoo5/ecg-reasoning-benchmark","repo_kind":"found_in_text","path":"ecg_reasoning_benchmark/utils.py","file_url":"https://github.com/Jwoo5/ecg-reasoning-benchmark/blob/HEAD/ecg_reasoning_benchmark/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e186589d35e07f11"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}