{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tablevqa-bench-a-visual-question-answering","title":"TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domains","arxiv_id":"2404.19205","date":"2024-04-30","proceeding":null,"authors":["Yoonsik Kim","Moonbin Yim","Ka Yeon Song"],"abstract":"In this paper, we establish a benchmark for table visual question answering, referred to as the TableVQA-Bench, derived from pre-existing table question-answering (QA) and table structure recognition datasets. It is important to note that existing datasets have not incorporated images or QA pairs, which are two crucial components of TableVQA. As such, the primary objective of this paper is to obtain these necessary components. Specifically, images are sourced either through the application of a \\textit{stylesheet} or by employing the proposed table rendering system. QA pairs are generated by exploiting the large language model (LLM) where the input is a text-formatted table. Ultimately, the completed TableVQA-Bench comprises 1,500 QA pairs. We comprehensively compare the performance of various multi-modal large language models (MLLMs) on TableVQA-Bench. GPT-4V achieves the highest accuracy among commercial and open-sourced MLLMs from our experiments. Moreover, we discover that the number of vision queries plays a significant role in TableVQA performance. To further analyze the capabilities of MLLMs in comparison to their LLM backbones, we investigate by presenting image-formatted tables to MLLMs and text-formatted tables to LLMs, respectively. Our findings suggest that processing visual inputs is more challenging than text inputs, as evidenced by the lower performance of MLLMs, despite generally requiring higher computational costs than LLMs. The proposed TableVQA-Bench and evaluation codes are available at \\href{https://github.com/naver-ai/tablevqabench}{https://github.com/naver-ai/tablevqabench}.","url_abs":"https://arxiv.org/abs/2404.19205v1","url_pdf":"https://arxiv.org/pdf/2404.19205v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tablevqa-bench-a-visual-question-answering","repo_url":"https://github.com/naver-ai/tablevqabench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.19205","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.19205"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/naver-ai/tablevqabench","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"7e854c1911105e72","entry":"evaluate_fintabnet","repo":"naver-ai/tablevqabench","repo_kind":"official","path":"src/evaluate_tablevqa_dataset.py","file_url":"https://github.com/naver-ai/tablevqabench/blob/HEAD/src/evaluate_tablevqa_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7e854c1911105e72"}},{"code_sha256_prefix":"c89aaf82328d2dc9","entry":"evaluate_tabfact","repo":"naver-ai/tablevqabench","repo_kind":"official","path":"src/evaluate_tablevqa_dataset.py","file_url":"https://github.com/naver-ai/tablevqabench/blob/HEAD/src/evaluate_tablevqa_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c89aaf82328d2dc9"}},{"code_sha256_prefix":"bc40b10cad1ef4d4","entry":"evaluate_wtq","repo":"naver-ai/tablevqabench","repo_kind":"official","path":"src/evaluate_tablevqa_dataset.py","file_url":"https://github.com/naver-ai/tablevqabench/blob/HEAD/src/evaluate_tablevqa_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"bc40b10cad1ef4d4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}