{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/chartqapro-a-more-diverse-and-challenging","title":"ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering","arxiv_id":"2504.05506","date":"2025-04-07","proceeding":null,"authors":["Ahmed Masry","Mohammed Saidul Islam","Mahir Ahmed","Aayush Bajaj","Firoz Kabir","Aaryaman Kartha","Md Tahmid Rahman Laskar","Mizanur Rahman","Shadikur Rahman","Mehrad Shahmohammadi","Megh Thakkar","Md Rizwan Parvez","Enamul Hoque","Shafiq Joty"],"abstract":"Charts are ubiquitous, as people often use them to analyze data, answer questions, and discover critical insights. However, performing complex analytical tasks with charts requires significant perceptual and cognitive effort. Chart Question Answering (CQA) systems automate this process by enabling models to interpret and reason with visual representations of data. However, existing benchmarks like ChartQA lack real-world diversity and have recently shown performance saturation with modern large vision-language models (LVLMs). To address these limitations, we introduce ChartQAPro, a new benchmark that includes 1,341 charts from 157 diverse sources, spanning various chart types, including infographics and dashboards, and featuring 1,948 questions in various types, such as multiple-choice, conversational, hypothetical, and unanswerable questions, to better reflect real-world challenges. Our evaluations with 21 models show a substantial performance drop for LVLMs on ChartQAPro; e.g., Claude Sonnet 3.5 scores 90.5% on ChartQA but only 55.81% on ChartQAPro, underscoring the complexity of chart reasoning. We complement our findings with detailed error analyses and ablation studies, identifying key challenges and opportunities for advancing LVLMs in chart understanding and reasoning. We release ChartQAPro at https://github.com/vis-nlp/ChartQAPro.","url_abs":"https://arxiv.org/abs/2504.05506v2","url_pdf":"https://arxiv.org/pdf/2504.05506v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"chartqapro-a-more-diverse-and-challenging","repo_url":"https://github.com/vis-nlp/chartqapro","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"chart-question-answering","task_name":"Chart Question Answering"},{"task_slug":"chart-understanding","task_name":"Chart Understanding"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2504.05506","atlas_url":"https://app.syntology.ai/?focus=2504.05506","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.05506"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vis-nlp/chartqapro","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fb35a4fb403a10b9","entry":"fix_list_format","repo":"vis-nlp/chartqapro","repo_kind":"official","path":"evaluate_predictions.py","file_url":"https://github.com/vis-nlp/chartqapro/blob/HEAD/evaluate_predictions.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fb35a4fb403a10b9"}},{"code_sha256_prefix":"36667e7b107acaec","entry":"load_predictions","repo":"vis-nlp/chartqapro","repo_kind":"official","path":"evaluate_predictions.py","file_url":"https://github.com/vis-nlp/chartqapro/blob/HEAD/evaluate_predictions.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"36667e7b107acaec"}},{"code_sha256_prefix":"af8755d4fa414194","entry":"parse_to_list","repo":"vis-nlp/chartqapro","repo_kind":"official","path":"evaluate_predictions.py","file_url":"https://github.com/vis-nlp/chartqapro/blob/HEAD/evaluate_predictions.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"af8755d4fa414194"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}